Method and apparatus for construction and operation of connected infrastructure

ABSTRACT

Augmented reality apparatus and methods of use are provided with persistent digital content linked to a location coordinates. More specifically, the present invention links a physical location with digital content to enable a user interface with augmented reality that combines aspects of the physical area with location specific digital content. According to the present invention, digital content remains persistent with a location even if visual aspects of the location change.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority as a continuation in part to NonProvisional U.S. patent application Ser. No. 17/244,970, filed Apr. 30,2021, and entitled METHOD AND APPARATUS FOR PERSISTENT LOCATION BASEDDIGITAL CONTENT; and also as a continuation in part to Non ProvisionalU.S. patent application Ser. No. 16/657,660, filed Oct. 18, 2019, andentitled METHOD AND APPARATUS FOR CONSTRUCTION AND OPERATION OFCONNECTED INFRASTRUCTURE Non Provisional patent application Ser. No.16/504,919, filed Jul. 8, 2019 and entitled METHOD AND APPARATUS FORPOSITION BASED QUERY WITH AUGMENTED REALITY HEADGEAR as a Continuationapplication; and as a Continuation application to Non Provisional patentapplication Ser. No. 16/503,878, filed Jul. 5, 2019 and entitled METHODAND APPARATUS FOR ENHANCED AUTOMATED WIRELESS ORIENTEERING; and as aContinuation application to Non Provisional patent application Ser. No.16/297,383, filed Jul. 5, 2019 and entitled SYSTEM FOR CONDUCTING ASERVICE CALL WITH ORIENTEERING; and as a Continuation application to NonProvisional patent application Ser. No. 16/249,574, filed Jan. 16, 2019and entitled ORIENTEERING SYSTEM FOR RESPONDING TO AN EMERGENCY IN ASTRUCTURE; and as a Continuation application to Non Provisional patentapplication Ser. No. 16/176,002, filed Oct. 31, 2018 and entitled SYSTEMFOR CONDUCTING A SERVICE CALL WITH ORIENTEERING; and as a Continuationapplication to Non Provisional patent application Ser. No. 16/171,593,filed Oct. 26, 2018 and entitled SYSTEM FOR HIERARCHICAL ACTIONS BASEDUPON MONITORED BUILDING CONDITIONS; and as a Continuation application toNon Provisional patent application Ser. No. 16/165,517, filed Oct. 19,2018 and entitled BUILDING VITAL CONDITIONS MONITORING; and as aContinuation application to Non Provisional patent application Ser. No.16/161,823, filed Oct. 16, 2018 and entitled BUILDING MODEL WITH CAPTUREOF AS BUILT FEATURES AND EXPERIENTIAL DATA; and as a Continuationapplication to Non Provisional patent application Ser. No. 16/142,275,filed Sep. 26, 2018 and entitled METHODS AND APPARATUS FOR ORIENTEERING;and to Provisional Patent Application Ser. No. 62/712,714, filed Jul.31, 2018 and entitled BUILDING MODEL WITH AUTOMATED WOOD DESTROYINGORGANISM DETECTION AND MODELING; and as a Continuation application toNon Provisional patent application Ser. No. 15/887,637, filed Feb. 2,2018 and entitled BUILDING MODEL WITH CAPTURE OF AS BUILT FEATURES ANDEXPERIENTIAL DATA; and as a Continuation application to Non Provisionalpatent application Ser. No. 15/716,133, filed Sep. 26, 2017 and entitledBUILDING MODEL WITH VIRTUAL CAPTURE OF AS BUILT FEATURES AND OBJECTIVEPERFORMANCE TRACKING; and as a Continuation application to NonProvisional patent application Ser. No. 15/703,310, filed Sep. 13, 2017and entitled BUILDING MODEL WITH VIRTUAL CAPTURE OF AS BUILT FEATURESAND OBJECTIVE PERFORMANCE TRACKING; and to Provisional PatentApplication Ser. No. 62/531,955, filed Jul. 13, 2017 and entitledBUILDING MODELING WITH VIRTUAL CAPTURE OF AS BUILT FEATURES; and toProvisional Patent Application Ser. No. 62/531,975, filed Jul. 13, 2017and entitled BUILDING MAINTENANCE AND UPDATES WITH VIRTUAL CAPTURE OF ASBUILT FEATURES; and to Provisional Patent Application Ser. No.62/462,347, filed Feb. 22, 2017 and entitled VIRTUAL DESIGN, MODELINGAND OPERATIONAL MONITORING SYSTEM. This application also claims priorityas a continuation in part to Non Provisional U.S. patent applicationSer. No. 17/196,146, filed Mar. 9, 2021, and entitled TRACKING SAFETYCONDITIONS OF AN AREA, which in turn is a continuation of NonProvisional U.S. patent application Ser. No. 16/935,857, filed Jul. 22,2020, and entitled TRACKING SAFETY CONDITIONS OF AN AREA; NonProvisional U.S. patent application Ser. No. 17/196,146 claims thebenefit of U.S. Provisional Patent Application 63/155,109 filed Mar. 1,2021 and entitled METHODS AND APPARATUS FOR WORKSITE MANAGEMENT BASEDUPON WIRELESS REAL TIME LOCATION AND DIRECTION; and also claims priorityas a continuation in part of Non Provisional U.S. patent applicationSer. No. 17/183,062 filed Feb. 23, 2021 and entitled METHODS ANDAPPARATUS FOR ENHANCED POSITION AND ORIENTATION BASED INFORMATIONDISPLAY; which in turn is a continuation of Non Provisional U.S. patentapplication Ser. No. 16/898,602 filed Jun. 11, 2020 and entitled METHODAND APPARATUS FOR ENHANCED POSITION AND ORIENTATION DETERMINATION. NonProvisional U.S. patent application Ser. No. 17/196,146 also claimspriority as a continuation in part of Non Provisional U.S. patentapplication Ser. No. 17/176,849 filed Feb. 16, 2021 and entitled METHODOF WIRELESS GEOLOCATED INFORMATION COMMUNICATION IN SELF-VERIFYINGARRAYS; which in turn is a continuation of Non Provisional U.S. patentapplication Ser. No. 16/915,155 filed Jun. 29, 2020 and entitled METHODOF WIRELESS DETERMINATION OF A POSITION NODE; which in turn claimspriority as a continuation of Non Provisional U.S. patent applicationSer. No. 16/775,223 filed Jan. 28, 2020 and entitled SPATIALSELF-VERIFYING ARRAY OF NODES. Non Provisional U.S. patent applicationSer. No. 17/196,146 also claims priority as a continuation in part ofNon Provisional U.S. patent application Ser. No. 17/134,824 filed Dec.28, 2020 and entitled METHOD AND APPARATUS FOR INTERACTING WITH A TAG INA COLD STORAGE AREA; which in turn claims priority as a continuation ofNon Provisional U.S. patent application Ser. No. 16/943,750, filed Jul.30, 2020 and entitled COLD STORAGE ENVIRONMENTAL CONTROL AND PRODUCTTRACKING. Non Provisional U.S. patent application Ser. No. 17/196,146also claims priority as a continuation in part of Ser. No. 17/113,368filed Dec. 7, 2020 and entitled APPARATUS FOR DETERMINING AN ITEM OFEQUIPMENT IN A DIRECTION OF INTEREST. Non Provisional U.S. patentapplication Ser. No. 17/196,146 also claims the benefit of U.S.Provisional Application Ser. No. 63/118,231 filed Nov. 25, 2020 andentitled METHODS AND APPARATUS FOR LOCATION BASED TRANSACTION SECURITY.Non Provisional U.S. patent application Ser. No. 17/196,146 also claimspriority as a continuation in part of Non Provisional U.S. patentapplication Ser. No. 16/951,550, filed Nov. 18, 2020 and entitled METHODAND APPARATUS FOR INTERACTING WITH A TAG IN A WIRELESS COMMUNICATIONAREA. Non Provisional U.S. patent application Ser. No. 17/196,146 alsoclaims the benefit of U.S. Provisional Application Ser. No. 63/093,416filed Oct. 19, 2020 and entitled METHOD AND APPARATUS FOR PERSISTENTLOCATION BASED DIGITAL CONTENT SECURITY. Non Provisional U.S. patentapplication Ser. No. 17/196,146 also claims the benefit of U.S.Provisional Application Ser. No. 63/088,527 filed Oct. 7, 2020 andentitled METHOD AND APPARATUS FOR PERSISTENT LOCATION BASED DIGITALCONTENT. Non Provisional U.S. patent application Ser. No. 17/196,146also claims priority as a continuation in part of U.S. Non Provisionalapplication Ser. No. 17/062,663 filed Oct. 5, 2020 and entitled METHODSAND APPARATUS FOR HEALTH CARE PROCEDURE TRACKING; which in turn claimspriority as a division to Non Provisional U.S. patent application Ser.No. 16/831,160 filed Mar. 26, 2020 and entitled METHODS AND APPARATUSFOR HEALTHCARE FACILITY OPTIMIZATION. The contents of each of which arerelied upon and incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to augmented reality with persistentdigital content linked to a physical location within various types ofinfrastructure such as bridges, dams, ports, roadways, and the like.More specifically, the present invention links location coordinatesdescriptive of a physical location with digital content to enable a userinterface with augmented reality that combines aspects of the physicalarea with location specific digital content. According to the presentinvention, digital content remains persistent with a location even ifvisual aspects of the location change.

BACKGROUND OF THE INVENTION

The use of smart devices has become a daily part of life for manypeople. A smart device may be used as a communication tool and toreceive information. Communication is generally point to point basis,and sometimes on a point to many points basis. Information disseminationto unknown recipients, or to recipients that are not ascertainable at atime of generation of the information, requires posting, or storing ofthe information in a centralized server where the information may beretrieved using traditional queries and search engines, such as Googleor social media.

However, it is difficult, if not impossible to automatically receiveinformation pertinent to a designated area. And it is especiallydifficult to receive information related to a subject area when a userdoes not know what to ask for, or what type of information may beavailable to the user. The ability to automatically receive informationpertinent to various forms of infrastructure with systems to support thedetermination of location and orientation of users to enable theretrieval and display of the information is needed.

SUMMARY OF THE INVENTION

Accordingly, the present invention combines methods and apparatus forproviding digital content included in an Agent interface based upon anarea at least partially defined by a geospatial position inferred by theAgent interface to support and display aspects of various types ofinfrastructure. In some embodiments, an area defined by a geospatialposition contemplated by an Agent is based upon a current positionoccupied by the Agent and a direction of interest provided by the Agentin the context of models of an infrastructure in the general location ofthe current position occupied by the Agent.

Digital content linked to a physical area is provided to an Agent basedupon an identification of the Agent, where the Agent is, and whichdirection the Agent is oriented towards. In some embodiments, thepresent invention provides an interactive user interface with contentderived from an immediate environment the Agent is located in andvirtual content. For example, the present invention enables a user todirect a Smart Device, such as a smart phone, towards an area ofinterest to the user and a user interface is presented on the user'ssmart device (and/or a remote smart device) the interface includes anaugmented reality environment that combines a rendition of the physicalenvironment present to the user and location specific information in theform of digital content.

For example, a transceiver may be co-located with a sensor and engage inwireless communication. Based upon the wireless communication, locationcoordinates indicating where the sensor is located. The sensorquantifies a condition at a specific location. When a user views thephysical area containing the sensor with the Smart Device, the SmartDevice displays a rendition of the area monitored by the sensor and withthe conditions quantified by the sensor.

Essentially, the present invention enables point and query (or ask andquery) access to information or other content close to a Smart Device.The Smart Device may be used to generate an interface indicating whatpeople, equipment, vehicles, or other items are viewable to the SmartDevice and place those items into the context of the environmentsurrounding the Smart Device.

In general, the present invention associates digital content with Tagsassociated with location coordinates. A Tag may include one or more of aPhysical Tag, a Virtual Tag, and a Hybrid Tag (as described in theGlossary below). Tags provide persistent access to specified digitalcontent based upon the Tag's association of the content with a set oflocation coordinates and/or an area of positional coordinates includingthe location coordinates. For example, a set of positional coordinatesmay be located near an architectural aspect of a structure, such as apoint of intersection of two beams. A Tag may be associated with the setof positional coordinates. An Area of digital content interaction (e.g.,retrieval of digital content and/or placement of digital content forsubsequent retrieval) may include the set of positional coordinates andalso include an area comprising additional sets of positionalcoordinates, surrounding the set of positional coordinates, adjacent tothe set of positional coordinates, and/or proximate to the set ofpositional coordinates.

This functionality is accomplished by establishing a target area anddetermining which tags are present within the target area. Tags may bevirtual; in which case the virtual tags are associated with positionalcoordinates and viewable whenever a target area is designated toencompass the coordinates the virtual tag.

Alternatively, the tags may be physical, such as a small disk adhered toan item of equipment, vehicle, or a person's employee badge. Tracking ofa position and content associated a physical tag may be updated in realtime or on a periodic basis. Physical tags may be moved into a targetarea or the target area may be moved to encompass the physical tag. Thepresent invention may automatically generate an interface indicatingwhich tags contained in the interface, what those tags are associatedwith and where a tag is in relation to the Smart Device. It may alsoaccess any information that has been stored and associated with the tagand present int on the Smart Device.

By aligning real world and virtual world content, a real world siteexperience is enriched with content from a geospatially linked virtualworld. The virtual world content is made available to an Agent basedupon a position and a direction of a Radio Target Area (“RTA”) specifiedby a Smart Device supported by the Agent. A geospatial position anddirection of interest that is contained within the RTA is generatedusing wireless communication with reference point transmitters. Wirelesscommunication capabilities of the Reference Point Transmitters determineparameters associated with a Wireless Communication Area (“WCA”). TheRTA is a subset of the WCA.

The present invention provides for methods and apparatus for executingmethods that augment a physical area, such as an area designated as awireless communication area. The method may include the steps oftransceiving a wireless communication between a Smart Device andmultiple reference point transceivers fixedly located at a positionwithin a wireless communication area; generating positional coordinatesfor the Smart Device based upon the wireless communication between theSmart Device and the multiple reference transceivers; establishing aradio target area for an energy receiving sensor; receiving energy intothe energy receiving sensor from the radio target area; generating adigital representation of the energy received into the energy receivingsensor at an instance in time; generating positional coordinates for atag at the instance in time, the tag comprising digital content andaccess rights to the digital content; determining the tag is locatedwithin the radio target area based upon the positional coordinates forthe tag; generating a user interactive interface comprising staticportions based upon the digital representation of the energy receivedinto the energy receiving sensor; generating a dynamic portion of theuser interactive interface based upon the positional coordinates for thetag and the positional coordinates for the Smart Device; receiving auser input into the dynamic portion of the user interactive interface;and based upon the user input received into the dynamic portion of theuser interactive interface, including the digital content in the userinteractive interface.

In some embodiments, multiple disparate energy levels may be receivedinto the energy receiving sensor at the instance in time, each disparateenergy level received from a different geospatial location; associatingpositional coordinates with the disparate energy levels; and indicatingthe disparate energy levels and relative positions of the disparateenergy levels in the user interactive interface. A tag may include avirtual tag with the digital content and a location identified viapositional coordinates.

In another aspect, a physical tag may include a transceiver capable ofwireless communication with the multiple reference transceivers and themethod may include transceiving a wireless communication between a tagand multiple reference point transceivers; and generating positionalcoordinates for the tag based upon the wireless communication betweenthe tag and the multiple reference transceivers. The wirelesscommunication between the Smart Device and the multiple reference pointtransceivers may be accomplished by transceiving using an Ultra-Widebandmodality; Bluetooth modality or other wireless modality, such as WiFi.

A wireless communication area may be identified as including a radiotransmission area of the energy receiving sensor and the wirelesscommunication area may be based upon a communication distance of theUltra-Wideband modality in an area encompassing the energy receivingsensor.

Transceiving a wireless communication between a tag and multiplereference point transceivers may be accomplished using w wirelessmodality such as, for example, a UWB or Bluetooth modality; andgenerating positional coordinates for the tag based upon the wirelesscommunication between the tag and the multiple reference transceiversmay be accomplished using the same modalities. Positional coordinatesmay include one or more of: Cartesian Coordinates, an angle of arrivaland an angle of departure and a distance.

In another aspect, access rights to tag content may be required andbased upon an identifier of the Smart Device or a user operating theSmart Device. A dynamic portion of the user interactive interface mayinclude an icon indicative of the digital content associated with thetag.

One general aspect includes a method for accessing digital contentassociated with a position in a wireless communication area in a regionof an infrastructure. The method also includes establishing a grossposition determination with a location sensor of a smart device;determining the region may include the infrastructure and the grossposition determination with an application program of the smart device;determining that the smart device has access to an array of referencepoint transceivers associated with the infrastructure and authorizingcommunications between the smart device and reference point transceiversof the array of reference point transceivers; transceiving a firstwireless communication between the smart device and a first referencepoint transceiver fixedly located at a first position within thewireless communication area; transceiving a second wirelesscommunication between the smart device and a second reference pointtransceiver fixedly located at a second position within the wirelesscommunication area; transceiving a third wireless communication betweenthe smart device and a third reference point transceiver fixedly locatedat a third position within the wireless communication area; generatingpositional coordinates for the smart device at an instance in time basedupon the first wireless communication, the second wirelesscommunication, and the third wireless communication, each wirelesscommunication between the smart device and a respective one of the firstreference point transceiver, the second reference point transceiver, andthe third reference point transceiver; generating a direction ofinterest from the positional coordinates of the smart device; generatingan area of interest may include the direction of interest; generatingpositional coordinates of a tag at the instance in time; associating thedigital content with the tag; determining that the positionalcoordinates of the tag are within the area of interest; determining thatthe smart device has access rights to the digital content; receiving auser input into a dynamic portion of a user interactive interface on thesmart device, the user input operative to cause the smart device todisplay the digital content; and based upon the user input received intothe dynamic portion of the user interactive interface, displaying thedigital content related to the infrastructure in the user interactiveinterface. Other examples of this aspect include corresponding computersystems, apparatus, and computer programs recorded on one or morecomputer storage devices, each configured to perform the actions of themethods.

Implementations may include one or more of the following features. Themethod where the location sensor utilizes global position system radiosignals to determine the gross position determination. Examples mayinclude those where the location sensor utilizes global position systemradio signals and cellular network signals to determine the grossposition determination. Examples may include those where the locationsensor utilizes cellular network signals alone to determine the grossposition determination. Examples may include those where the array ofreference point transceivers may include more than three reference pointtransceivers. Examples may include those where the array of referencepoint transceivers forms a self-verifying array of nodes. Examples wherethe tag is a physical tag may include a sensor measuring at least afirst physical attribute of the infrastructure. The digital content mayinclude an historical record of measurements of the first physicalattribute. A server may include the digital content performs algorithmiccalculations upon the measurements of the first physical attribute togenerate a first message to a user as a portion of the digital content.The first message to the user provides directions for the user to moveto a fourth position, where at the fourth position the user records asecond measurement with an energy receiving sensor of the smart deviceto record an image of a state of the infrastructure. In some examples,the infrastructure is a bridge.

In some examples, the tag is a virtual tag, where the virtual tagassociates a communication node with the infrastructure. The digitalcontent may include persistent data associated with the virtual tag. Thedigital content may include a second message from a second user to userswho gain access to the virtual tag. The smart device may include aconnected sensor device, where the connected sensor device produces ameasurement of a second physical attribute of the infrastructure andwhere the smart device communicates data associated with the measurementof the second physical attribute as persistent data to be storedassociated with the virtual tag. A server which may include the digitalcontent may perform algorithmic calculations upon the measurements ofthe second physical attribute to generate a second message to the useras a portion of the digital content. The second message to the userprovides directions for the user to move to a fifth position, where atthe fifth position the user records a third measurement with an energyreceiving sensor of the smart device to record an image of a state ofthe infrastructure. The infrastructure is a bridge. The user ispresented with a first user interface of the smart device, where the tagis presented as an icon overlaid on an image generated based on theposition and the direction of interest of the user. The first userinterface presents a set of icons each icon at the set of iconspresented at a respective location value of both virtual tags andphysical tags in the region of the infrastructure.

The details of one or more examples of the invention are set forth inthe accompanying drawings and the description below. The accompanyingdrawings that are incorporated in and constitute a part of thisspecification illustrate several examples of the invention and, togetherwith the description, serve to explain the principles of the invention:other features, objects, and advantages of the invention will beapparent from the description, drawings, and claims herein.

DESCRIPTION OF THE DRAWINGS

The accompanying drawings, that are incorporated in and constitute apart of this specification, illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention:

FIG. 1 illustrates location determination with wireless communication toreference points.

FIG. 2 illustrates locations aspects with sonic location.

FIGS. 2A-2B illustrate an example of construction sites and userlocation and orientation.

FIGS. 2C-2H,2J-2M illustrates examples of infrastructure and relatedaspects.

FIGS. 3-3A illustrate methods of orienteering by device movement.

FIGS. 4A-4D illustrate exemplary configurations of antenna arrays.

FIG. 5 illustrates an exemplary Smart Device with an array of antennas.

FIG. 6 illustrates exemplary methods of indicating directions with SmartDevices and antenna arrays.

FIG. 7 illustrates an exemplary method of a user utilizing an orientedstereoscopic sensor system to orient a direction of interest.

FIGS. 8A-8G illustrate aspects of the determination of directions ofinterest and Fields of View and information display.

FIGS. 9A-9C illustrate additional aspects of information display.

FIGS. 10A-10B illustrates an exemplary method for generating anaugmented-reality Radio Target Area for a Smart Device.

FIG. 11 illustrates an exemplary database structure according to theinstant specification.

FIG. 12 illustrates additional exemplary method for displaying RadioTarget Areas with Smart Devices.

FIG. 13 illustrate exemplary aspects of Wireless Communication Areas inRadio Target Area display.

FIG. 14 illustrates a set of polygons generated via LIDAR that may beused for geospatial recognition.

FIG. 15 illustrates apparatus that may be used to implement aspects ofthe present disclosure including executable software.

FIG. 15A illustrates apparatus that may be used to implement aspects ofthe present invention including executable software.

FIG. 15B illustrates an exemplary block diagram of a controller withangle of arrival and angle of departure functionality.

FIG. 16A illustrates an exemplary handheld device that may be used toimplement aspects of the present disclosure including executablesoftware.

FIGS. 16 B-D illustrates exemplary aspects of nodes that may be used toimplement aspects of the present disclosure including executablesoftware.

FIG. 17 illustrates exemplary structure of handheld smart devices usedto implement aspects of the present disclosure.

FIGS. 18A-D illustrate exemplary aspects of object persistence invirtual and real world environments.

FIGS. 19, 19A-19H, 19J illustrate exemplary aspects related to variousexamples of infrastructure.

DETAILED DESCRIPTION

The present invention relates to Agent interfaces such as an augmentedreality interface with persistent digital content linked to a physicallocation. More specifically, the present invention links a physicallocation with digital content to enable a user interface with augmentedreality that combines aspects of the physical area with locationspecific digital content. According to the present invention, digitalcontent remains persistent with a location even if visual aspects of thelocation change. Methods and apparatus are provided for determining theexistence of digital content linked to positional coordinates; anddisplaying real-world energy levels integrated with and aligned withvirtual-world digital content.

In the following sections, detailed descriptions of examples and methodsof the invention will be given. The description of both preferred andalternative examples though thorough are exemplary only, and it isunderstood that, to those skilled in the art, variations, modifications,and alterations may be apparent. It is therefore to be understood thatthe examples do not limit the broadness of the aspects of the underlyinginvention as defined by the claims.

In some embodiments, the location of a Node. may be determined viadiscernment of a physical artifact, such as, for example a visuallydiscernable feature, shape, or printed aspect. A pattern on a surfacemay convey a reference point by a suitable shape such as a cross,Vernier or box structure as non-limiting examples. The printing may alsoinclude identification information, bar codes or lists of locationcoordinates directly. A Smart Device ascertaining a physical referencemark and a distance of the Smart Device to the mark may determine arelative location in space to a coordinate system of the marks.

Marks tied to a geospatial coordinate system may be utilized todetermine a relative location. A number of methods may be executed todetermine a distance from the Smart Device to a mark such as, forexample, a sensed reflection of light beams (preferably laser beams),electromagnetic beams of wavelength outside of the visible band such asIR, UV, radio and the like, or sound-based emanations. It may beimportant that the means of determining the distance can be focused intoa relatively small size. It may be important that the means ofdetermining the distance is reflected by the physical mark. For example,a light-source means of determining the distance may benefit from amirror surface upon the physical mark. And it may be important that thereflected signal emerges significantly towards the user. It may bedesirable that the physical reference points are placed with highaccuracy at specific reference locations, or it may be desirable to beable to measure with high accuracy the specific reference locationsafter placement.

In some examples, a carefully placed reference point Node may functionas a transceiver of signals. For example, a Node may receive andtransmit signals in a radio frequency band of the electromagneticspectrum. In a simple form, a Node may detect an incoming signal andbroadcast a radio frequency wireless communication. Frequencies utilizedfor wireless communication may include those within the electromagneticspectrum radio frequencies used in UWB, Wi-Fi, and Bluetooth modalities,as well as IR, visible and UV light as examples.

In some embodiments, sound emanations may also be used as acommunication mechanism between a Smart Device and a reference pointNode. In some examples, the Nodes may function to communicate data withtheir electromagnetic or sonic transmissions. Such communications mayprovide identifying information unique to the Node, data related to thesynchronization of timing at different well located reference points andmay also function as general data communication Nodes. A triangulationcalculation and/or a distance and angle indicating a position of a SmartDevice or a Node may result from a system of multiple reference positionNodes communicating timing signals to or from the Smart Device or Node.Methods of calculating positions via wireless communications may includeone or more of: RTT, RSSI, AoD, AoA, timing signal differential and thelike. Triangulation or other mathematical techniques may also beemployed in determining a location.

A process is disclosed for determination of a position based upontriangulation with the reference points may be accomplished, for examplevia executable software interacting with the controller in a SmartDevice, such as, for example via running an app on the Smart Device.

Reference points may be coded via identifiers, such as a UUID(Universally Unique Identifier), or other identification vehicle.Referring now to FIG. 1, aspects of a system for enhanced wirelessposition and orientation are illustrated. Reference Point Transceivers101-104 are shown deployed within or proximate to (in this case withinradio transceiving distance) a Structure 106, wireless communicationsbetween the reference point transceivers 101-104 and a Transceiver 105supported an Agent 100 within or proximate to the Structure 106 may beused to calculate a local position of the Transceiver 105 supported anAgent 100. In this discussion one or more of: a Transceiver 105supported an Agent 100; and a reference point transceivers 101-104 maybe embodied as a Node; a Smart Device; or a dedicated transceiver.

A designated direction 112 may be generated based upon wirelesscommunications and may be used to determine a targeted item 113 and/or atargeted direction. A designated direction may be determined via thewireless modalities discussed herein. Some embodiments will include oneor more of: radio communications involving multiple antennas; radiocommunications with one or more antennas at two or more locations atdifferent instances of time; a magnetometer; a compass; LiDAR; laser;sonic; accelerometer; gyroscope or other electronic mechanism.

A Radio Target Area 117 (sometime abbreviated as “RTA”) may beidentified with an area of from which energy is received from adirection congruent with and/or overlapping an area indicated with thedesignated direction 112. In some embodiments, an RTA may also becongruent and/or overlap a field of view (FOV) of a camera deviceincluded in the Smart Device at the location of transceiver 105.

In another aspect, in some embodiments, one or more of a physicaltransceiver, such as one or more of: an IoT Tag 114; a Virtual Tag 115;and a Hybrid Tag 116. In general, an IoT Tag 114 includes a physical tagwith a transceiver that transceives logical data that may include atiming signal from which a position may be calculated. In someembodiments, an IoT Tag 114 may also include an electronic sensor thatgenerates digital content that may also be transceived and/or stored inthe IoT Tag 114. A Virtual Tag includes digital content associated witha set of coordinates but does not include a physical transceiver. AHybrid Tag 116 includes digital content associated with a set ofcoordinates similar to a Virtual Tag 115 and may be a subset type ofVirtual Tag 115 that includes digital content generated via an IoT Tag114, while the IoT Tag 114 was located at the coordinates associatedwith the Hybrid Tag 116.

In some embodiments, one or more of the IoT Tag (Virtual Tag 115), theSmart Device at the location of transceiver 105, and the Reference PointTransceivers 101-104 may comprise a Node with a transceiver, acontroller, and a memory that may store digital content.

Still further, in some embodiments, a Node deployed as one or more ofthe IoT Tag (Virtual Tag 115), the Smart Device at the location oftransceiver 105, and the Reference Point Transceivers 101-104 mayreceive a digital content download via a wireless modality other thanthe nodality used to determine a geospatial location 107 of the SmartDevice at the location of transceiver 105, such as, for example via acellular transmission or a satellite download in a remote area, such asa remote construction work site, a wilderness location, a hiking trail,a remote section of an aqueduct or utility or a pipeline (e.g. a gas oroil pipeline). Digital content downloaded and/or stored in a Nodedeployed as one or more of the IoT Tag (Virtual Tag 115), the SmartDevice at the location of transceiver 105, and the Reference PointTransceivers 101-104 may be communicated during a transceiving processengaged in between Nods and/or device described herein.

Reference Point Transceivers 101-104 may be fixed in a certain locationwithin or proximate to the Structure 106 and define a wirelesscommunication area 111. The Reference Point Transceivers 101-104 maytransceive in a manner suitable for determination of a position of oneor more Transceivers 105 supported by an Agent 100. Transceiving may beconducted via one or more wireless transmission modalities between theTransceiver 105 supported by the Agent 100 and one or more ReferencePoint Transceivers 101-104.

By way of non-limiting example, Transceivers 105 supported by the Agent100 may be included in, and/or be in logical communication with, a SmartDevice, such as, one or more of: a smart phone, tablet, headgear, ring,watch, wand, pointer, badge, Tag, Node; a Smart Receptacle; headgear; orTag supported by the Agent 100 or other Agent supportable device withTransceiver 105 operable to transceive with the Reference PointTransceivers 101-104.

The Reference Point Transceivers 101-104 may include devices, such as,for example, a radio transmitter, radio receiver, a light generator, alight receiver, a pattern recognizing device, a sonic or ultrasonicdevice. A radio frequency transceiver may transmitters and receiversoperative to communicate via wireless modalities such as, for example:Wi-Fi, Bluetooth, Ultra-wideband (“UWB”), ultrasonic, infrared, or othercommunication modality capable of logical communication betweenTransceivers 101-105.

In some embodiments, a Reference Point Transceivers 101-104 may includea multi-modality transceiver, that communicates more locally via a firstcommunication modality and a second communication modality. Each of thefirst modality and the second modality, may include for exampleapparatus and software code for wireless communications via one of: UWB,Bluetooth, Wi-Fi, ANT, Zigbee, BLE, Z Wave, 6LoWPAN, Thread, Wi-Fi,Wi-Fi-ah, NFC (near field communications), Dash 7, Wireless HART, orsimilar modality. In some embodiments, a first communication modalitymay engage in wireless communications with a first set of distanceconstraints and a second communication modality may engage in wirelesscommunication with a second set of distance constraints. For example, afirst communication modality may involve UWB communications modalitiesthat travel a shorter distance than a second communication modality butmay provide a more accurate location determination. In this example, asecond communication modality may include one or more of: satellitecommunications (e.g., GPS); cellular communication modalities (e.g., 3G,4G 5G and the like), sub GHz communications, or another modality.

In some embodiments, satellite communications (e.g., global positioningsystem “GPS”) and/or cellular communications may be used to determine anidentification of a structure 106 within which transceiving with theReference Point Transceivers 101-104 may be established.

Wireless communications between Transceivers 101-105 may engage inlogical communications to provide data capable of generating one or moreof: Cartesian coordinates, polar coordinates, vector values, AoA, AoD,RTT, RSS, a GPS position, or other data that may be utilized for one ormore of: locating one or both of an Agent 100; indicating a direction ofinterest; and identify a Structure or defined area of structure 106.Logical communications may include digital data indicative or a value,which may be used, for example, to determine a distance via processesinvolving one or more of: a time difference of arrival (TDOA); frequencydifference in arrival (FDOA); and time of flight (TOF).

A precise geospatial location 107 may be determined via triangulationbased upon a measured distance from three or more Reference PointTransceivers 101-104. For example, a radio transmission or light signalmay be measured and compared from the three reference point transceivers101-103. Other embodiments may include a device recognizable via imageanalysis and a sensor or other Image Capture Device, such as a CCDdevice, may capture an image of three or more Reference PointTransceivers 101-104. Image analysis may recognize the identification ofeach of three or more of the Reference Point Transceivers 101-104 and asize ratio of the respective image captured with Reference PointTransceivers 101-104 may be utilized to calculate a precise position.Similarly, a height designation may be made via triangulation using thereference point transceivers as reference to a known height or areference height.

Transceivers 101-105 may include circuitry and logic capable oftransceiving in a single modality, or multiple disparate modalities.Similarly, a Reference Point Transceivers 101-104 and/or anAgent-supported device at the location of Transceiver 105 may includemultiple transceiver device, including, transmitters and receivers.

A modality, as used in conjunction with a Transceiver, transmitterand/or receiver refers to one or both of a bandwidth of wirelesscommunication and a protocol associated with a bandwidth. By way ofnon-limiting example, a modality, as used in relation to a Transceiver,transmitter and/or receiver may include Wi-Fi; Wi-Fi RTT; Bluetooth;UWB; Ultrasonic, sonic, infrared; ANT, Zigbee, BLE, Z Wave, 6LoWPAN,Thread, Wi-Fi, Wi-Fi 33-ah, NFC (near field communications), sub-GHz,Dash 7, Wireless HART, or other logical communication medium.

Triangulation essentially includes determining an intersection of threedistances 108-110, each of the three distances 108-110 calculated from areference point transceivers 101-104 to an Agent-supported device at thelocation of transceiver 105. The presence invention allows for a firstdistance 108, to be determined based upon a wireless communication in afirst modality; and a second distance 109 and a third distance 110determined based upon a wireless communication in a same or differentmodality as the first modality. For example, a first distance 108 may bedetermined based upon a wireless communication using UWB; a seconddistance 109 may be determined based upon a wireless communication usingBluetooth; and a third communication may be determined based upon awireless communication using ultrasonic communication (othercombinations of same and/or different communication modalities are alsowithin the scope of the present invention and may include, for example,infrared communications, image recognition, RFID, accelerometer readingsor other data generated by an electronic device or mechanicalmechanism).

A geospatial location 107 may be determined via triangulation based upona measured distance from three or more reference point transceivers101-103 to the transceiver 105 supported by the Agent 100. For example,timing associated with a radio transmission or light signal may bemeasured and compared from the three reference point transceivers101-103. Other embodiments may include a device recognizable via imageanalysis and a sensor or other Image Capture Device, such as a CCDdevice, may capture an image of three or more reference pointtransceivers 101-104.

Additional embodiments may include image analysis of image data capturedvia a CCD included in a Smart Device to recognize the identification ofeach of three or more of the reference point transceivers 101-104 and asize ratio of the respective image captured where reference pointtransceivers 101-104 may be utilized to calculate a precise position.Similarly, a height designation may be made via triangulation using thereference point transceivers as reference to a known height or areference height. In a similar fashion, triangulation may be utilized todetermine a relative elevation of the Smart Device as compared to areference elevation of the reference points.

In some embodiments, the geospatial location 107 of the Agent-supporteddevice at the location of Transceiver 105 may be ascertained via one ormore of: triangulation; trilateration; and multilateration (MLT)techniques.

In some embodiments, a geospatial location based upon triangulation maybe generated based upon a controller receiving a measurement of anglesbetween the position and known points at either end of a fixed baseline.By way of non-limiting example, a point of a geospatial location may bedetermined based upon generation of a triangle with one known side andtwo known angles. Moreover, a geospatial location based uponmultilateration may be generated based on a controller receivingmeasurement of a difference in distance to two reference positions, eachreference position being associated with a known location. Wirelesssignals may be available at one or more of: periodically, withindetermined timespans and continually. The determination of thedifference in distance between two reference positions provides multiplepotential locations at the determined distance. A controller may be usedto generate a plot of potential locations. In some embodiments, thepotential determinations generally form a curve. Specific embodimentswill generate a hyperbolic curve.

The controller may be programmed to execute code to locate a relativelyexact position along a generated curve, which is used to generate ageospatial location. The multilateration system thereby receives asinput multiple measurements of distance to reference points, wherein asecond measurement taken to a second set of stations (which may includeone station of a first set of stations) is used to generate a secondcurve. A point of intersection of the first curve and the second curvemay be used to indicate a specific location.

In another aspect, in some embodiments, the location of a Transceiver101-105 may be determined and/or aided via discernment of data basedupon a physical artifact, such as, for example a visually discernablefeature, shape or printed aspect located within the Structure 106.Discernment of the physical artifact may, for example, be based upontopographical renditions of physical aspects included in the Structure,such as those measured using LIDAR, a magnetic force, image data (or apoint cloud derived from image data). A pattern on a surface may conveya reference point by a recognizable pattern (which may be unique to thesetting), Vernier or three-dimensional structure as non-limitingexamples. A Smart Device ascertaining a physical reference mark and adistance of the Smart Device to the mark may determine a relativelocation in space to a coordinate system of the marks.

Marks and/or markers with a known location indicated via coordinatevalues tied to a geospatial coordinate system may be utilized todetermine a relative location. Recognition of the marker may beaccomplished for example, via one or more of image recognition, laserreflection, RFID activation and reading, sonic reading, LiDAR, or otherrecognition technology. Similarly, number of methods and apparatus maybe executed in various embodiments, to determine a distance from theSmart Device to a mark such as, for example, a sense reflection of lightbeams (preferably laser beams), electromagnetic beams of wavelengthoutside of the visible band such as IR, UV, Radio and the like, orsound-based emanations.

In some examples, a carefully placed reference point Node may functionas a transceiver of signals. For example, a Node may receive andtransmit signals in a radio frequency band of the electromagneticspectrum. In a simple form, a Node may detect an incoming signal andcoincidently broadcast a radio frequency wireless communication.Frequencies utilized for wireless communication may include those withinthe electromagnetic spectrum radio frequencies used in UWB, Wi-Fi, andBluetooth modalities, as well as IR, Visible and UV light as examples.

In some embodiments, sound emanations may also be used as acommunication mechanism between a smart device and a Reference PointTransceivers 101-104. In some examples, the Reference Point Transceivers101-104 may function to communicate data with their electromagnetic orsonic transmissions. Such communications may provide identifyinginformation unique to the Node, data related to the synchronization oftiming at different well located reference points and may also functionas general data communication nodes. A triangulation calculation of theposition of a Smart Device or a Node may result from a system ofmultiple reference position Nodes communicating timing signals to orfrom the Smart Device or Node. Methods of calculating positions viawireless communications may include one or more of: RTT, RSSI, AoD, AoA,timing signal differential and the like, Triangulation or othermathematical techniques may also be employed in determining a location.

The process of determination of a position based upon triangulation withthe reference points may be accomplished, for example via executablesoftware interacting with the controller in a Smart Device, such as, forexample via running an app on the Smart Device.

Reference points may be coded via identifiers, such as a UUID(Universally Unique Identifier), or other identification vehicle.

Reference point transceivers 101-104 may be deployed in a defined areaof structure 106 to determine a geospatial location 107 of an Agent 100within or proximate to the defined wireless communication area 111.Reference point transceivers 101-104 may be fixed in a certain locationand transceive in a manner suitable for a triangulation determination ofthe position of the Agent. Transceiving may occur via wirelesstransmission to one or more Transceivers 105 supported by the Agent 100.By way of non-limiting example, Transceivers 105 supported by the Agent100 may be included in, or be in logical communication with, a SmartDevice with Transceivers 105 able to transceive with the reference pointtransceivers 101-104.

The reference point transceivers 101-104 may include devices such as aradio transmitter, radio receiver, a light generator, or animage-recognizable device (i.e., an apparatus set out in a distinctivepattern recognizable by a sensor). A radio transmitter may include a UWBNode, Wi-Fi, Bluetooth, or other communication device for entering intological communication between Transceivers 101-105. In some embodiments,Reference Point Transceivers 101-104 may include a Wi-Fi router thatadditionally provides access to a distributed network, such as theInternet. Cartesian coordinates (including Cartesian coordinatesgenerated relative to a GPS or other reference point), or any othercoordinate system, may be used as data that may be utilized for one ormore of: locating one or both of an Agent 100; indicating a direction ofinterest; and identifying a Structure 106 or defined area of structure106. A radio transmitter may include a router or other Wi-Fi device. Theradio transmitter may include transmissions via an Ultra-Wideband(“UWB”) frequencies including, for example, 3.5-6.5 GHz; on Wi-Fifrequencies (300 MHz-60 GHz), sub GHz frequencies or another modality. Alight generator may distribute light at human-safe intensities and atvirtually any frequency known in the art. Such frequencies include,without limitation, infrared, ultraviolet, visible, or nonvisible light.Further, the light beacon may comprise a laser, which may transmit lightat any of the aforementioned frequencies in a coherent beam.

This plurality of modalities allows for increased accuracy because eachmodality may have a different degree of reliability. For example, aSmart Device 101 and/or Smart Receptacle may measure a timing signaltransmitted by a Reference Point Transceivers 101-104 within a differenterror tolerance than it may measure the receipt into a photodetector ofinfrared laser light. This has at least two principal benefits. First, alocation calculation may, in some embodiments, be a weighted average ofthe location calculated from each modality. Second, outliers may beshed. For example, if the standard location calculation comprises aweighted average of the location as calculated by five modalities, butone modality yields a location greater than two standard deviations fromthe average computed location, then that modality may not be consideredfor future weighted location calculations.

Additionally, the radio transmitters and/or transceiver in the SmartDevice may comprise multiple antennas that transmit signals in astaggered fashion to reduce noise. By way of non-limiting example, ifthere are three antennas, then they may transmit a signal in intervalsof 20 milliseconds. Given this rate of transmission, a detected time ofarrival may be used to determine the distance between the transmitterand the antenna (i.e., the Smart Device). Moreover, the antennas maycomprise varying lengths to accommodate desirable wavelengths. Further,dead reckoning may be used to measure location, using traditionalmethods of numerical integration.

A precise location may be determined based upon wireless transmissionsbetween Nodes, such as between a Smart Device and the Reference PositionTransceivers. Timing determinations—as well as signal qualities likeangle of arrival, angle of departure, transmission strength,transmission noise, and transmission interruptions—may be considered ingenerating relative positions of Nodes. Additional considerations mayinclude AI and unstructured queries of transmissions between Nodes andtriangulation logic based upon a measured distance from three or moreReference point transceivers 101-104. For example, a radio transmissionor light emission may be measured, and timing associated with the radiotransmission or light to determine a distance between Nodes. Distancesfrom three reference point transceivers 101-103 may be used to generatea position of a Node in consideration. Other methodologies includedetermination of a distance from one or more Nodes and a respectiveangle of arrival and/or angle of departure of a radio or lighttransmission between the Node in consideration and another Node(Reference Point Node or dynamic position Node).

In some embodiments of the invention, position determination in aStructure or on a Property contemplates determination of a geospatiallocation using triangulation, trilateration, or multilaterationtechniques. In some embodiments, a geospatial location relative to oneor more known reference points is generated. The geospatial location inspace may be referred to as having an X, Y position indicating a planardesignation (e.g., a position on a flat floor), and a Z position (e.g.,a level within a Structure, such as a second floor) may be generatedbased upon indicators of distance from reference points. Indicators ofdistance may include a comparison of timing signals received fromwireless references. A geospatial location may be generated relative tothe reference points. In some embodiments, a geospatial location withreference to a larger geographic area is associated with the referencepoints, however, in many embodiments, a controller will generate ageospatial location relative to the reference point(s) and it is notrelevant where the position is located in relation to a greatergeospatial area. In addition to these Cartesian coordinates, polarcoordinates may be used, as further described below.

A geospatial location based upon triangulation may be generated basedupon a controller receiving a measurement of angles between the positionand known points at either end of a fixed baseline. A point of ageospatial location may be determined based upon generation of atriangle with one known side and two known angles.

Triangulation essentially includes determining an intersection of threedistances 108-110, each of the three distances 108-110 calculated from areference point transceivers 101-104 to an Agent-supported device at thelocation of transceiver 105. The present invention allows for a firstdistance 108 to be determined based upon a wireless communication in afirst modality; and a second distance 109 and a third distance 110determined based upon a wireless communication in a same or differentmodality as the first modality. For example, a first distance 108 may bedetermined based upon a wireless communication using UWB; a seconddistance 109 may be determined based upon a wireless communication usingBluetooth; and a third communication may be determined based upon awireless communication using ultrasonic communication (othercombinations of same and/or different communication modalities are alsowithin the scope of the present invention).

Geospatial location 107 may be based upon triangulation generated basedupon a controller receiving a measurement of angles between the positionand known points at either end of a fixed baseline. A point of ageospatial location may be determined based upon generation of atriangle with one known side and two known angles.

In some embodiments, a geospatial location 107 may be based uponmultilateration and generated based on a controller receiving ameasurement of a difference in distance to two reference positions, eachreference position being associated with a known location. Wirelesssignals may be available at one or more of: periodically, withindetermined timespans, and continually. The determination of thedifference in distance between two reference positions provides multiplepotential locations at the determined distance. A controller (such asone in the Smart Device) may be used to generate a plot of potentiallocations. In some embodiments, the potential determinations generallyform a curve. Specific embodiments will generate a hyperbolic curve.

A controller may be programmed to execute code to locate an exactposition along a generated curve, which is used to generate a geospatiallocation. The multilateration thereby receives as input multiplemeasurements of distance to reference points, wherein a secondmeasurement taken to a second set of stations (which may include onestation of a first set of stations) is used to generate a second curve.A point of intersection of the first curve and the second curve is usedto indicate a specific location.

In exemplary embodiments, as described herein, the distances may betriangulated based on measurements of UWB, Wi-Fi or sub GHz strength attwo points. Transceiver signals propagate outward as a wave, ideallyaccording to an inverse square law. Ultimately, a crucial feature of thepresent invention relies on measuring relative distances between twopoints. In light of the speed of Wi-Fi waves and the real-timecomputations involved in orienteering; these computations need to be ascomputationally simple as possible. Thus, depending upon a specificapplication and mechanism for quantifying a condition or location, suchas a measurement, various coordinate systems may be desirable. Inparticular, if the Smart Device moves only in a planar direction whilethe elevation is constant, or only at an angle relative to the ground,the computation less complicated.

One exemplary coordinate system includes a polar coordinate system. Oneexample of a three-dimensional polar coordinate system is a sphericalcoordinate system. A spherical coordinate system typically comprisesthree coordinates: a radial coordinate, a polar angle, and an azimuthalangle (r, θ, and φ, respectively, though θ and φ are occasionallyswapped conventionally).

By way of non-limiting example, suppose Point 1 is considered the originfor a spherical coordinate system (i.e., the point (0, 0, 0)). For eachTransceiver 101-105, emitter e₁, e₂, e₃, . . . can be described aspoints (r₁, θ₁, φ_(i)), (r₂, θ₂, φ₂), and (r₃, θ₃, φ₃) . . .respectively. Each of the r_(i)'s (1≤i≤3 or more) represent the distancebetween the Transceiver 101-105 emitter and the Transceiver 101-105receiver on the Smart Device 101 or Smart Receptacle (see FIG. 5A).

It is understood that in some embodiments, an azimuth may include anangle, such as a horizontal angle determined in an arcuate manner from areference plane or other base direction line, such as an angle formedbetween a reference point or reference direction; and line (ray orvector) such as a ray or vector generated from or continuing to a SmartDevice. In preferred embodiments, the ray or vector may be generallydirected from a Reference Position Transceiver towards, and/or intersectone or more of: an item of interest; a point of interest; anarchitectural aspect (such as a wall, beam, header, corner, arch,doorway, window, etc.); an installed component that may act as areference in an augmented virtual model (AVM) (such as, for example, anelectrical outlet, a light fixture, a plumbing fixture, an architecturalaspect; an item of equipment; an appliance; a multimedia device, etc.);another Reference Position Transceiver or other identifiabledestination.

Accordingly, in some embodiments, a spherical coordinate system mayinclude Reference Position Transceiver that is capable of determining anangle of departure of a location signal and a Transceiver that iscapable of determining an angle of arrival of the location signal; oneor both of which may be used to facilitate determination of anapplicable azimuth.

According to various embodiments of the present invention, one or bothof an angle of departure and an angle of arrival may therefore beregistered by a Transceiver that is transmitting and/or receivingwireless signals (e.g., radio frequency, UWB, Bluetooth 5.1, sonicfrequency, or light frequency).

In some embodiments, locating an Agent occurs in or proximate to aStructure in which Reference point transceivers, (including, forexample, one or more of: Wi-Fi Transceivers, UWB Transceivers, BluetoothTransceivers, infrared Transceivers, and ultrasonic Transceivers) may belocated above and/or below an Agent. In these embodiments, a cylindricalcoordinate system may be more appropriate. A cylindrical coordinatesystem typically comprises three coordinates: a radial coordinate, anangular coordinate, and an elevation (r, θ, and z, respectively). Acylindrical coordinate system may be desirable where, for example, allWi-Fi emitters have the same elevation. Angles may be determined asdescribed above.

In some embodiments, transceivers 101-105 including arrays of antennasmay be used to measure an angle of radio communication (e.g., angle ofarrival and/or angle of departure). Various configurations oftransmitting antennas and receiving antennas may be used. For example, aradio transmission may be transmitted with a single antenna and receivedwith a receiver with an array of antennas, the phase or timingdifference of arriving signals can be used to calculate the angle atwhich the signals emerged. In angle of departure schemes, a transmittermay contain an array of antennas and may send a pattern of signalsthrough the array that arrive at a receiver with a single antenna wherethe angle of departure (AoD) is communicated.

Measurement of angle of arrival may be performed as mentioned bycalculation of time difference of arrival at the antennas in an array oralternatively can be performed by rotation of antenna elements.

Some modalities, such as those modalities that adhere to the Bluetooth5.1 or BLE5.1 standards, allow a Smart Device 101, Smart Receptacle, orother Node to determine an angle of arrival (AoA) or an angle ofdeparture (AoD) for a wireless transmission. An array of antennas may beused to measure aspects of the Bluetooth signaling that may be useful tocalculate these AoA and AoD parameters. By calibrating an antennasystem, the system may be used to determine angles in one or twodimensions depending on the design of the antenna. The result may besignificant improvement in pinpointing the location of origin of asignal.

An array of antennas may be positioned relative to each other and atransmitting transceiver to allow for extraction of an AoA/AoD. Such anarray may include a rectangular array; a polar or circular array; alinear array; and a patterned array, where a number of antennas aredeployed in a pattern conducive to a particular environment fortransceiving. Antennas may be separated by characterized distances fromeach other, and in some examples, a training protocol for the antennaarray results in antenna positioning incorporating superior angle andlocation precision. Some transceivers may transceive in 2.4-2.482 GHzfrequency bands, and thus the radiofrequency transmissions may havewavelengths in the roughly 125 mm length scale. A collection of antennasseparated by significantly less than the wavelength may function bycomparing a phase of RF transmissions arriving at the antennas. Anaccurate extraction of phase differences can yield a difference in pathlength that, when accumulated, can lead to a solution for the anglesinvolved. In some embodiments, Transceivers 101-105 may include antennaarrays combined with batteries and circuitry to form completeself-contained devices.

In an example, an Agent-supported device at the location of transceiver105 may be located at a position and may transmit a signal of thevarious types as have been described. Nodes, such as Reference PointTransceivers 101-104 located at reference points in the wirelesscommunication area around the position of the Agent 100 may receive thetransmission and determine the angle of arrival of that transmission.Similarly, transmission associated with other reference pointtransceivers 101-103 may also determine an angle of arrival oftransmissions. In some embodiments, reference point transceivers 101-103may communicate with one or more of: each other, a smart device, acontroller, or other processor to mathematically process multiple anglesand locations of the transceivers and calculate a position of atransmission emanation. In examples where calculations are not performedat a smart device, a communication to the smart device of the calculatedposition may be communicated.

In certain embodiments of the invention, a direction of interestindicated by Smart Device 101 or a Smart Receptacle (see FIG. 5A) may bedetermined based upon a movement of Smart Device 101 or a SmartReceptacle 502. For example, a device with a controller and anaccelerometer, such as mobile Smart Device, may include an Agent displaythat allows a direction to be indicated by movement of the device from adetermined location acting as a base position towards an As Builtfeature in an extended position. In some implementations, the SmartDevice may first determine a first position based upon triangulationwith the reference points and a second position (extended position) alsobased upon triangulation with the reference points. These positiondeterminations may proceed as described above. The process ofdetermination of a position based upon triangulation with the referencepoints may be accomplished, for example via executable softwareinteracting with the controller in the Smart Device, such as, forexample via running an app on the Smart Device. Logical communicationsrelevant to location determination may include, for example, one or moreof: timing signals; SIM information; received signal strength; GPS data;raw radio measurements; Cell-ID; round trip time of a signal; phase; andangle of received/transmitted signal; time of arrival of a signal; atime difference of arrival; and other data useful in determining alocation.

In another aspect, captured data may be compared to a library of storeddata using image recognition software to ascertain and/or affirm aspecific location, elevation and direction of an image capture locationand proper alignment with the virtual model. Similarly, a Smart Devicemay capture image data and/or audio data from a determined position andin some embodiments, a determined direction. The captured image data maybe stored in a library of image data and referenced in subsequentanalysis of where a location is.

In various embodiments, a position of an Agent Smart Device may bedetermined by any of the methods described herein. A user or other Agentmay position a sensor of an associated smart device to be pointing in adirection of interest and obtain an image. The image may be passed on toa server with access to database of images containing stored images ofthe space around the user. Algorithms on the server may compare thestored images to the image captured by the user and may calculateadjustments to the comparative image based on where the reference imagewas taken in relationship to the location of the user. Based on thedetermination that the calculated adjusted image compared to the imageobtained by the user in the direction of interest, a direction may beinferred with known location of objects in the reference image. In somevariations, the differences in features of the user obtained imagecompared to a reference image may be used to calculate a direction ofinterest based upon a location at which the reference image wasobtained.

In some examples, stored images may be obtained at multiple angles toimprove accuracy of orienteering. These examples may include sensorarrays, audio capture arrays and sensor arrays with multiple datacollection angles. In some examples a full 360-degree sensor perspectivemay be obtained by such arrays. In some directional arrays, a Sensorarray (including image capture sensors) may include at least 120 degreesof data capture. By collecting such image collections as theSensor/Sensor systems are moved, a database of image perspectives may beformed and utilized to assist in orienteering as described.

Non-limiting examples may include image-based identification where adevice with some imaging means, including but not limited to a mobiledevice sensor, tablet device sensor, computer sensor, security sensor,or A/R headset sensor, may image points of interest in a direction ofinterest. These points of interest may be identified. Image recognitionsoftware may be used to identify the visualized landscape by itsidentifying features. Machine learning may be used to train systemsusing this software to identify specific features of the environment inquestion.

To create a supplemental topographic part of a model of the environmentof a user, laser scanning and/or LiDAR may be performed during theacquisition of imagery for a reference database. A resultingthree-dimensional shape model may be modelled with the captured imageryto help in the comparison to user data. Three-dimensional shapes can beused to infer comparative imagery at different angles of acquisitionthan exist in a database. In another example, a device of a user mayhave the means of performing laser scanning or LiDAR scanning of theenvironment as well as obtaining images. The resultant three-dimensionaldata or a composite of the three-dimensional data, and the imagery maybe used to recognize features and determine a direction that the userwas facing when they collected the image.

The results of scanning may be stored and presented in differentmanners. In some examples, the scanned data may be represented by apoint cloud representation; in other examples an estimated topographicsurface representation may be used to visualize the three-dimensionalshape data obtained. In some examples, outward facing planes of thesurface topography may have the captured imagery superimposed upon them.The resulting image and three-dimensional models may be used tocalculate a direction of interest or a device field of view in a dynamicsense or alternatively upon user request.

In some examples other methods of capturing spatially accurateinformation may include the use of drones and optical scanningtechniques which may include high resolution imagery obtained frommultiple viewpoints. Scanning may be performed with light based methodssuch as a CCD sensor. Other methods may include infrared, ultraviolet,acoustic, and magnetic and electric field mapping techniques.

In other embodiments, a single distance to a point of interest in animage, which may be obtained by a laser, other collimated light source,sound source or the like, may be used with models of the environment ofthe user. A comparison of the imagery and the measurement of thedistance of the user to a prominent feature in the image may allow foran orientation of the user to be determined algorithmically.

In some exemplary embodiments, radio frequencies used for wirelesscommunication include sound waves used to perform one or more of:location determination; movement tracking in interior or exterior areas;and distance calculation. Sound wave transmissions include a number ofsignificant attributes, which may translate into a benefit for a givenset of circumstances when used for RF based location determination.

According to the present invention, sonic waves may be deployedindependently, or in combination with electromagnetic transmissions andreception of logical communications utilizing other bandwidths, such asbandwidths associated with Ultrawideband, Wi-Fi, Bluetooth,Ultrawideband, ANT, infrared or almost any wavelength in the Industrial,Scientific, and Medical bands (sometimes referred to as “ISM Bands”).

For example, sound waves travel through an ambient atmosphere at asignificantly slower speed than electromagnetic radiation(6×10{circumflex over ( )}2 m/sec versus 3×10{circumflex over ( )}8m/sec). Therefore, a relative accuracy for measurements related totravel times may be improved in some environments by orders of magnitudeusing sonic-based location tracking as compared to electromagnetic-basedmeasurements. Therefore, using sonic communications may result inincreased accuracy of location determination, in some environments.

The present invention also provides for sonic wave emanations may beused to complement electromagnetic emanations based upon a tendency thatsonic waves generally do not efficiently penetrate walls other physicalitems or structures. Sonic transceivers may be particularly advantageousin a defined area where location can be unambiguously determined to bewithin a particular room (the use of multiple bandwidth transmitting andreceiving for different purposes is further discussed below). Sound waveinteraction with a solid surface, such as a wall, may require that foroptimal performance, transceiver/transmitters/receivers to be located ineach room where location detection is desired. In some embodiments, areflected sonic transmission may be received and analyzed to determinean effect of the reflected nature of the transmission.

Accordingly, methods may be employed using sonic emanations andreception for location determination. In general, frequencies ofeffective indoor sonic location detection may be at ultrasonicbandwidths (commonly used bandwidths include frequencies of between 1MHz and 10 MHz, although frequencies of less than 50 kHz to greater than200 MHz are possible). The utilized frequencies may be either below orabove audible detection by people or other animals in the location, suchas at frequencies above 20 kHz.

Sound waves may be used to perform one or more of: locationdetermination, movement tracking in interior or exterior locations, anddistance calculation from a position to a Smart Device at the locationof transceiver 105, which may be accomplished based upon transmissionand receipt of sonic transmission. Sound wave transmissions includeseveral significant attributes, which may be beneficial for a given setof circumstances when used for radiofrequency-based locationdetermination. According to the present invention, sonic waves may bedeployed independently, or in combination with, transmissions andreception of logical communications utilizing other bandwidths, such asbandwidths associated with Wi-Fi, Bluetooth, ANT, infrared or almost anywavelength in the ISM Bands.

In some examples, as may be used in orienteering herein, anAgent-supported device at the location of transceiver 105 may supportreceivers, transmitters or transceivers that interact with ultrasonictransceivers fixedly secured to a reference point position, such as viamechanical mounting within a room environment. An ultrasonic positioningsystem may have indoor positioning accuracy at centimeter, millimeter,and even sub-millimeter accuracy. Multiple ultrasonic Transceivers maytransceive from an Agent-supported device to communicate with fixedreference point transceivers may transmit signals. Arrival of the soundtransmissions may be accurately timed and converted to distances. Insome embodiments, distance determinations may be improved with knowledgeof temperatures in the environment containing the sound transceiving.For example, temperature may be measured at one or more of: anAgent-supported Smart Device, a Reference Point position, and an ambientenvironment.

In some examples, synced timing apparatus is able to generate a locationof a stationary Agent to within centimeter accuracy using sonic wavetransmissions and reception and preferably within several millimeters ofaccuracy. In addition, in some embodiments sensors are able to detectfrequency shifts within the sonic emanations which may add informationabout the relative rate of movement of the Agent, which may then in turnallow for correction to the timing signals.

In some examples, a combination of radio frequency emissions andultrasonic emissions may be used. For example, a complement of radiofrequency emissions/receptions and ultrasonic of radio frequencyemissions and ultrasonic emissions may be reconciled to generate moreaccurate location determination. In another aspect, a radio frequencysignal may be used to transmit syncing signals that establish a timethat ultrasonic signals are transmitted. Since, the electromagnetictransmissions may be orders of magnitude faster than soundtransmissions, the electromagnetic transmissions relatively small timeof travel from the Transceivers to the Agent may be negligible andtherefore used as “zero-time” setpoints as received at theAgent-supported Transceiver. In such embodiments, a controllerdetermining a location may use not only relative arrival times, but alsoa delta time between a radiofrequency transmission and ultrasonictransmission to determine a distance from a transmitting Transceiver. Anarray of such ultrasonic and/or radiofrequency transceivers provideincreased accuracy in triangulating a location of the Agent.

In still further examples, RF communications may transmit a syncingpulse and also transmit digital data about various aspects of a definedarea, such as the defined area's identification, its relative and/orabsolute location in space and other refinements. In some examples, datarelated to improved distance calculation may also be transmitted by RFcommunication such as temperature of the environment, humidity and thelike which may influence the speed of sound in the environment as anon-limiting example. In some examples, such a system may result inmillimeter level accuracy of position determination.

In some examples, the process may be iterated to refine the direction ofeach of the ultrasonic transmitters and maximize signal levels of thetransmission which may provide additional information in the calculationof a position. UWB, Bluetooth, Wi-Fi, ANT, Zigbee, BLE, Z Wave, 6LoWPAN,Thread, Wi-Fi, Wi-Fi-ah, NFC (near field communications), Dash 7, and/orWireless HART transmissions may be used for data communications andsyncing. In other examples, an Agent-supported device at the location oftransceiver 105 may be moved and an iterative process may be used totrack the Agent-supported device at the location of transceiver 105 asit moves through space. Stationary Agents may be tracked withsubmillimeter accuracy in some embodiments.

A direction dimension may be based upon multiple transceivers includedin a Smart Device or a Smart Receptacle or via a movement of a SmartDevice or Smart Receptacle while an Agent supporting the Smart Device orSmart Receptacle remains in a stationary position in relation toreference, such as a ground plane position. For example, a device with acontroller and an accelerometer, such as mobile Smart Device, mayinclude a user display that allows a direction to be indicated bymovement of the device from a determined location acting as a baseposition towards a feature in the intended direction where the movementresults in an extended position. In some implementations, the SmartDevice may first determine a first position based upon triangulationwith the reference points and a second position (extended position) alsobased upon triangulation with the reference points.

As described above, facing a mobile device towards an area in aStructure and movement of the mobile device in a particular pattern maybe used to ascertain a specific area in space to be interpreted byvarious methods. In some examples, the leading edge of a smart device,or the top portion of the user screen (in either portrait or landscapemode of display) may be the reference for the direction pointed in bythe user. If the smart device is held at an angle relative to theground, in some examples, the angle formed by the perpendicular to thetop portion of the user screen may be projected upon the ground and thatprojection taken as the indication of direction.

Referring now to FIG. 2, a secure transaction system 200 with apparatusand methods of completing a transaction with multifactor security isillustrated. A Smart Device 205 is illustrated proximate to aTransaction Apparatus 208. The methods and apparatus discussed hereinmay be used to determine that the Smart Device 205 is located within aSmart Device Position Area 207. In some embodiments, a Smart DevicePosition Area 207 is generally defined as an area in front of (or to aparticular side of) a Transaction Apparatus 208 within a margin ofaccuracy for wireless communication modalities and environmentalconditions (including, for example, environmental interference withwireless communication).

An Authorized Transaction Area (ATA) 209 may be specified to indicate anarea from which a Smart device 205 is authorized to perform transactionswith the Transaction Apparatus 208, In some embodiments, an ATA 209 maybe exclusive to a particular Transaction Apparatus 208. For example, asdiscussed further below, an ATA 209 may be exclusive to transactionswith a specified Transaction Apparatus 208, such as, for example, aparticular workstation; dispensing apparatus; building delivery; utilitystation; point of sale device; and construction site under management.

Inclusion of a Smart device 205 within an ATA 209 may be determinedaccording to any of the methods and apparatus discussed herein, inaddition a Transaction Apparatus Transceiver (TAT) 210 may be positionedwith the Transaction Apparatus 208 (such as, co-located with, orincorporated into the Transaction Apparatus 208). The TAT 210 mayinclude a relatively shorter range and more accurate location modalitythan GPS, such as one or more of: UWB, Bluetooth, Wi-Fi, ANT, Zigbee,BLE, Z Wave, 6LoWPAN, Thread, Wi-Fi, Wi-Fi-ah, NFC (near fieldcommunications), Dash 7, and/or Wireless HART transmissions. The TAT 210may provide data sufficient to calculate an ATA/Smart Device distance206 indicative of how far the Smart Device at the location oftransceiver 105 is from the TAT 210 based upon a timing signal includedin transceiving between the TAT 210 and the Smart Device at the locationof transceiver 105.

In general, a TA/Smart Device distance 206 may not be enough todetermine that the Smart Device at the location of transceiver 105 islocated within a transaction Apparatus 208 since a radio distance isoften a radius. Therefore, the present invention provides for a TAT 210to provide for directional transmissions. For example, a directionaltransmission may be limited to transmission from a front side of thetransaction Apparatus 208 in a forward direction (as indicated by thearrow).

In addition, one or more Reference Point Transceivers 201-204 may engagewireless communications with the Smart Device 205 and a respectivedistance between the respective reference point transceivers 101-104 andthe Smart Device 205 may be calculated. Since only one respectivedistance 201-202 per reference point transceivers 101-104 will intersecta radius of a distance from a TAT 210, a determination made be made asto whether a SD is within an ATA 209.

With a Smart Device 205 positioned within an ATA 209, a transaction withimproved security may be entered into and completed. A number of factorsmay be ascertained before a transaction with the transaction Apparatus208 is executed. In some preferred embodiments, the completion of thetransaction with the transaction Apparatus 208 will include one or moreof: a) designating a user via a unique user identification; b) exchangeof a user password, private key, synchronous dynamic password or othersecurity code; c) location verification via wireless communication,which may include determination that the Smart Device at the location oftransceiver 105 is within a ATA 209; identifying a transaction Apparatus208; d) present transaction Apparatus 208 credentials to a transactionprocess; f) present credentials to a transaction process; g) designateactions to be completed during transaction execution; h) execute theactions; generation of an augmented reality icon viewable form the ATA209; and interaction with the A/R icon.

According to the present invention, presentation of credentials from oneor both of the transaction apparatus 208 and Agent 100 may only be madeonce location determinations indicate that the Agent 100 with the UserSmart Device at the location of transceiver 105 is within the ATA 209.

In addition, in some embodiments, credentials may be withheld until anA/R Virtual Tag icon has been generated and made available to the uniqueidentifier of the user. The Virtual Tag icon will be located in virtualspace at a geospatial position that is viewable via an A/R interfacegenerated with input from geospatial indicators generated by the SmartDevice. The viewable area may include a Radio Target Area for an energyreceiving device in (or attached to) the Smart device. Geospatialindicators may include, for example, location coordinates for the SmartDevice that are generated as discussed herein. Generation of the icon ina virtual space that is aligned with a specific location creates adeterrent to unauthorized access to the Transaction Apparatus since aperson that is not present to the TA will not be able to view theVirtual Tag icon and therefore not be able to interact with the icon inorder to complete a process for gaining access to the securetransaction.

Referring now to FIGS. 2A-2B exemplary embodiments of apparatus andmethods that may be involved in various embodiments involving VirtualTags 211, 211A-211B according to the present invention are illustrated.These illustrations are non-limiting and depict only some specificexamples of how the present invention may be implemented to provideenhanced security and data sources to memorializing events and/or foranalysis by machine learning apparatus, such as via those apparatuscapable of artificial intelligence processes.

Referring nor to FIG. 2A, a workstation 212 is illustrated as aTransaction Apparatus for which a geospatial component is required tologin to the workstation 212 or to otherwise operate the workstation.For example, a Smart Device at the location of transceiver 105 may berequired to be within an ATA 209A located in an area proximate to theworkstation 212. In this scenario, the term “proximate to” may be adefined radius distance from the workstation, such as for example,within 3 feet, 6 feet, 12 feet, or other defined distance. “Proximateto” may also include a directional component, such as, for example, infront of the workstation 212 in order for an Agent 100 to login to theworkstation 212 and operate the workstation. An ATA 209A may also be alarger area such as an area within a structure in which the workstation212 resides, or a property in which the workstation 212 resides.

In some embodiments, a presence of the Smart Device at the location oftransceiver 105 within an ATA 209A permits an Agent 100 to enter Usercredentials. In other embodiments, the locating of the Smart Device atthe location of transceiver 105 within the ATA 209A causes a firstcontroller (such as a CPU in a workstation 212) to execute a securityrelated process that presents User credentials associated with the SmartDevice at the location of transceiver 105 to a related login process.Presentation of credentials may also be a part of other various securetransactions that require different credentials to be presented. Inaddition, credentials from one or both of a TA, such as the workstation212, and an Agent 100 may be presented to a process that willtransacted.

In some embodiments, a Virtual Tag 211A is presented at locationcoordinates within an RTA (not illustrated). The Virtual Tag 211A ispreferably viewable from within the ATA 209A with the Smart Device atthe location of transceiver 105, when the Smart Device at the locationof transceiver 105 is supported by the Agent 100.

Referring now to FIG. 2B, an exemplary worksite is illustrated withvarious items of equipment 213A-213C, personnel 214A-214C, roboticAgents 215, materials 216, authorized areas of work 217, and locationsof architectural aspects 218, such as a building, walkway, stairs etc.

According to the present invention, a personnel, which in someembodiments may also be a User 214A may locate a Smart Device at thelocation of transceiver 105 with an ATA 209B and interact with an App(or other executable code executed by a controller) that identifies aparticular authorized area of work 217 which may also represent a TAbased upon the Smart Device at the location of transceiver 105 beinglocated in the ATA 209B. The App may present Agent credentials for acontroller, software, automation, or other provider of access to VirtualTag(s) 211,211A-211B.

The App may also present payor credentials for the User 214A (or otherentity that authorizes payment for sale of materials, energy, services,or other saleable item or quantity to the User 214A or an entityassociated with the User 214A).

In some preferred embodiments, the credentials are virtually presentedto a known payment processor, such as a bank, a credit card company,online payment company (e.g., Venmo, PayPal, Zelle etc.) via a secureInternet connection or device specific app, thereby adding furthersecurity features. In this manner, the present invention provides for apayee for a transaction from within a specific TA for one or more of: a)a finite list of potential purchases; b) a finite list of potentialvendors to purchase from; c) a capped amount of purchase contingent uponthe Agent identification and the TA the Agent is in; d) unlimitedpurchase conditions with each purchase associated with the particularproject or other accounting designation, wherein the accountingdesignation is associated with a particular TA; and f) unlimitedpurchase conditions and/or accounting variables based upon the TA andAgent credentials.

For example, a User 214A in an ATA 209B may execute an app that providesa user interface with user interactive devices enabling one or more of:the purchase, shipping, supply. lease and rental of one or more of:materials; services; utilities; equipment; supplies; food; space; realestate; or other purchasable item and have the purchase linked with ajobsite associated with one or more of: an ATA 209B; an architecturalaspect 218; and a User 214A-214B. No card swipe, chip scan, near fieldcommunication or other locally interceptable transfer of data takesplace; and the point of sale may be anywhere within an ATA 209B.

In some embodiments, an additional step may be included in a securetransaction according to the present invention. The additional step mayinclude generation of a Virtual Tag 211 in RTA viewable from the ATA209B. Use of the Virtual Tag 211B verification requires that a User beable to see the Virtual Tag 211B and interact with the Virtual Tag 211B.In some embodiments, a Virtual Tag 211B may occupy a congruent space ina user interface as an item in a visual representation (e.g., image datareproducing a physical area). The item may be, by way of non-limitingexample, an item of equipment 213A-213C; personnel 214A-C; robotic agent215; materials 216, authorized area 216-219; or other ascertainable itemin a visual representation included in a user interface presented on aSmart Device at the location of transceiver 105.

A remote hacker or a hacker that was not in or very close to the ATA209B would not be able to view and interact with the Virtual Tag 211Bsince the Virtual Tag 211B is assigned to a set of location coordinatesthat are unpublished to the User 214A except as an icon in an A/Rinterface. Similar processes may be utilized in other securetransactions that may, for example, involve access to protected digitalcontent and submission of protected digital content.

In some embodiments, an action taken, or quantification of a conditionon the worksite may be accomplished via a robotic Agent 215, such as anUAV or UGV, in such embodiments an alternate ATA 209B may be designatedas an area for a specific action and an icon may be locatable by therobotic Agent 215, as well as by a User 214A overseeing the roboticAgent 215. The icon may include digital content related to an action tobe performed by the robotic device 215 at a particular location. Forexample, the icon may include a type of material to be used; a fastenertype; parameters of installation, time of installation; sequence ofinstallation; values for variables related to installation (e.g., force,pressure, torque, thickness of material, rate of process used, etc.).

In another aspect, in some embodiments, a secure transaction may includeone or more of: operation of various items of equipment 213A-213C;operation of a device 219, such as an automated lock that provides entryinto a designated area or structure (such as a delivery area, equipmentcorral, commercial building, garage, parking area, recreation facilityor fenced in area); association of digital content with a position of adesign model, site plan, floorplan or other AVM; and access to a utilityarea. Utility areas may include, for example, an electrical powersubstation, a gas line control area, water supply area, wastewater area,Internet or other distributed network area, watershed area, or otherutility area. A secure transaction may include a delivery, or simpleaccess to the area via an ATA 209B. As with other secure transactionsaccording to the present invention, a Virtual Tag Icon may be generatedproximate to the ATA 209B (in this case viewable from the ATA 209B) suchthat User 214A interaction with the generated icon may be used tofurther verify an authorized User 214A is present at the designated areaallocated as the ATA 209B. A utility area may also be equipped with anATA 209C accessible via UAV and/or UGV.

While a construction site is illustrated, other outdoor, or combinedoutdoor and structure interior environments are within the scope of thepresent invention. Accordingly, a definable geospatial area, such as arecreation area; hunt club area; private land; Federal Bureau LandManagement area, National Park, State, or local park, including landareas and/or water body areas may be designated to contain one or moreATA's 209B. One or more ATA's 209B may be associated with one or moreVirtual Tag Icons to verify a User 214A and a location that may bereferenced in an automated process for authorizing a transaction, whichmay include authorization to be present on the land included as amanaged area, a delivery, a movement of items within the area, or otheraction. Areas that contain wildlife and/or human activity may alsorequire authorization for activities monitoring movement of the wildlifeand/or humans and/or equipment.

There may also be numerous types of non-building infrastructure that aredefinable in a geospatial area and are within the scope of the presentinvention. Referring to FIGS. 2C-2M, various types of infrastructurewhich may be modelled to create an AVM and to which sensing devices maybe placed are illustrated. The sensed values may be included as layersin the AVM and may drive functionality of the modelling system asdiscussed previously and in following sections. Agents may arrive at aninfrastructure site and may access the AVM in the various mannersdiscussed and interact for access to various data and sensor readingsand the like.

Referring to FIG. 2C, an exemplary bridge infrastructure 220C isillustrated. A structural model of the bridge infrastructure such as asuspension bridge 220C may be installed in an infrastructure AVM. Theinfrastructure AVM may include associations of structural components andmay include various sensors located on respective structural components.Examples of said structural components may include one or more of:Suspender Cables 222C, Bridge Decks 221C, Towers 223C, Main CableConnectors 225C, and Main Cables 224C and the like. In some examples,the sensors may include operation capable of one or more of: generatingempirical quantification of an amount of vibration, a seismometermeasurement, a strain measurement, temperature, humidity, precipitation,wind speed, wind force, wind direction, relative distances, criticalpoint spatial locations, traffic load, traffic counts, camera feeds,generating empirical quantification of an amount of movement, such assettling or movement with underlying earth, as well as numerous otherempirical quantification of a condition that may relate to an ability ofa structure to perform successfully during deployment for a particularpurpose.

Deployment may include, by way of non-limiting example, one or more of:a bridge system 220C supporting motor vehicle traffic; railroad traffic;bicycle traffic, or foot traffic by humans and/or animals.

Referring to FIG. 2D, an exemplary dam infrastructure 220D isillustrated. The structural model of the dam or a retention dike may beinstalled in the AVM and may reference various sensor locations. Sensorsmay be deployed at locations in or around the dam that are capable ofproviding empirical measurements of conditions relevant to operation of,and/or a condition of the dam. Accordingly, the present invention mayinclude a Retained Water sensor 221D, Release Sluice top sensor 222D,Release Sluice bottom sensor 223D, soil sensors 224D located adjacent tothe dike and/or Dike Walls 225D. In some examples, the sensors 221D-225Dmay be functional to empirical data for conditions including, one ormore of: water levels, water pressures, vibrations, seismometermeasurements, strain measurements, temperature, humidity, precipitation,wind speed, wind force, wind direction, relative distances, criticalpoint spatial locations, camera feeds as well as numerous other sensing.

Referring to FIG. 2E, an exemplary railroad infrastructure 220E isillustrated with infrastructure sensors 221E-225E. An AVM may include avirtual structural model of the railroad infrastructure 220E installedin the AVM such that the AVM may reference various sensor locations andempirical quantification of conditions present at respective sensorlocations at a particular time and date. Examples of sensors deployed inrelation to a railroad infrastructure may include a train sensor 221E, afirst Railroad Tie sensor 222E (located at a first location), a secondRailroad Tie sensor 223E located at a second location, a first RailroadTrack sensor 224E located at a first railroad track location, and asecond Railroad Track sensor 225E located at a second railroad tracklocation. In some examples, the sensors may be functional to providequantification of one or more of: track weight, track locations insectpresence, vibrations, seismometer measurements, strain measurements,temperature, humidity, precipitation, wind speed, wind force, winddirection, relative distances, critical point spatial locations, camerafeeds as well as numerous other sensing.

As discussed earlier, in some embodiments of the present invention, afirst sensor, such as a first Railroad Track sensor 224E or a firstRailroad Tie sensor 222E may be used to introduce, first point in timeT1 a vibration pattern into a structural component of the railroadinfrastructure 200E with one or more of: a known frequency, a knownamplitude, and a known pattern into a first portion of a railroadinfrastructure a proximate position and a second sensor, such as asecond Track sensor 225E or a first Railroad Tie sensor 223E may be usedto receive a resultant vibration at a distal position at a second timeT2. In addition, T1 and T2 may be associated with a date of thevibration being introduced and received.

Each introduction of a vibration pattern at a first positional point maybe compared to a respective receipt of a vibration at a secondpositional point and differences in the vibration pattern may beanalyzed to determine an ability of an infrastructure, such as railroadinfrastructure, to perform a deployment task, such as support a train ormotor vehicle traversing the infrastructure 220E, 226E. This method ofintroduction and receiving of a vibration may be adapted for use withmultiple types of infrastructure, such as those illustrated in FIGS.2C-2M and discussed herein.

Referring to FIG. 2F, an exemplary motor vehicle roadway infrastructureis illustrated. For the purposes of this disclosure a motor vehicleroadway infrastructure may include one or more of: a highway, a parkway,a freeway, a state road, a county road, a private roadway, or othersurface designated for, and suitable for travel by a motor vehicle, suchas an automobile, a truck, a tractor trailer, a motorcycle, or othervehicle generally suitable for registration with a state for operationon a public roadway. The structural model of the roadway may beinstalled in the AVM and may reference various sensor locations such ason a Highway, on Middle Roadway Sensors 1&2 227AF and 227BF, on RoadwayLane Sensors 228AF and 228BF, Roadway Side Sensors 229AF and 229BF, andthe like. As discussed above, sensors 227AF-BF, 228AF-BF, 229AF-BF maybe operative to provide empirical quantification of different physicalconditions present in an infrastructure, such as a roadwayinfrastructure 226F.

In addition, sensors 227AF-BF, 228AF-BF, 229AF-BF may includetransceivers capable of receiving and/or transmitting wirelesscommunications. The wireless communications may include, for examplelocation information, identifying information, an infrastructurecondition, sensor status or other logical com Bluetooth communication.

In some embodiments, sensor transmissions may be useful to communicatewith a transceiver fixed to a vehicle on or near the roadwayinfrastructure. Positional calculations based upon the position ofsensors and a relative position of a vehicle may be of sufficientaccuracy to enable the vehicle to automatically maintain a proper pathof movement on the roadway infrastructure. Some embodiments mayadditionally include one or both of an angle of departure and an angleof arrival between a vehicle and a sensor 227AF-BF, 228AF-BF, 229AF-BFlocated within or near to a roadway infrastructure, may be registered bya transceiver that is transmitting and/or receiving wireless signals(e.g., radio frequency, sonic frequency, or light frequency).

Referring to FIG. 2G, an exemplary utility conduit infrastructure isillustrated. The structural model of the utility conduit may beinstalled in the AVM and may reference various sensor locations such asa Tunnel 230G, an Outer Tunnel Wall 231G, an Inner Tunnel Wall 232G, aTunnel Ceiling 233G, Utility Mediums 234G and Tunnel Vents 235G asnon-limiting examples. In some examples, the sensors may include sensingof utility flows, utility pressures, vibrations, seismometermeasurements, strain measurements, temperature, humidity, relativedistances, critical point spatial locations, camera feeds as well asnumerous other sensing.

Referring to FIG. 2H, an exemplary aqueduct infrastructure isillustrated. The structural model of the aqueduct may be installed inthe AVM and may reference various sensor locations such as an ElevatedWaterway 240H, Water 241H, a first Elevated Waterway Wall 242H, a secondElevated Waterway Wall 243H, a third Elevated Waterway Wall 244H, anElevated Waterway Stanchion 245H, and an Elevated Waterway Dock 246H asexamples. In some examples, the sensors may include sensing of waterlevels, water pressures, water flow rate, vibrations, seismometermeasurements, strain measurements, temperature, humidity, precipitation,wind speed, wind force, wind direction, boat traffic counts, relativedistances, critical point spatial locations, camera feeds as well asnumerous other sensing.

Referring to FIG. 2J, an exemplary waterway infrastructure isillustrated. The structural model of the waterway may be installed inthe AVM and may reference various sensor locations such as a CanalSystem 250J, a Canal Sidewall 251J, adjacent Soil to a Canal 252J, aCanal Bridge Support 253J, and the water of a Canal 254J as examples. Insome examples, the sensors may include sensing of water levels, waterpressures, water flow rate, vibrations, seismometer measurements, strainmeasurements, temperature, humidity, precipitation, wind speed, windforce, wind direction, boat traffic counts, relative distances, criticalpoint spatial locations, camera feeds as well as numerous other sensing.

Referring to FIG. 2K, an exemplary hydropower generator infrastructure260K is illustrated. The structural model of the hydropower generatormay be installed in the AVM and may reference various sensor locations.For example, the locations may include the hydropower generatorinfrastructure 260K, a Power Station 261K, a Hydroelectric Dam 262K, aHydroelectric Sluice 263K, the water supply for the Hydroelectricgenerator 264K, and Downstream Water locations such as on a tetheredbuoy as non-limiting examples. In some examples, the sensors may includesensing of water levels, water pressures, water flow rate, vibrations,seismometer measurements, strain measurements, temperature, humidity,precipitation, wind speed, wind force, wind direction, electricaloutput, relative distances, critical point spatial locations, camerafeeds as well as numerous other sensing.

Referring to FIG. 2L, an exemplary nuclear power generatorinfrastructure is illustrated. The structural model of the nuclear powergenerator may be installed in the AVM and may reference various sensorlocations. Examples of these locations may include a Nuclear Power Plant270L, on the site there may also be a Cooling Tower 271L, a Generator272L, Piping 273L, and positions of Ingress/Egress 274L as examples. Insome examples, the sensors may include sensing of radiation levels,cooling water levels, cooling water pressures, cooling water flow rate,cooling water temperatures, water bacterial level, vibrations,seismometer measurements, strain measurements, temperature, humidity,precipitation, wind speed, wind force, wind direction, electricaloutput, relative distances, critical point spatial locations, camerafeeds as well as numerous other sensing.

Referring to FIG. 2M an exemplary coal/gas power or Waste recyclinggenerator infrastructure is illustrated. The structural model of thegenerator may be installed in the AVM and may reference various sensorlocations. Examples of such locations may include a Waste Disposal Plant280M, adjacent soil to a generator plant 281M, a generator plant ExhaustStack 282M, a Disposal Facility 283M, and a source of cooling wateradjacent to a generator 284M. In some examples, the sensors may includesensing of ambient gas measurements, ambient particulate levels, coolingwater levels, cooling water pressures, cooling water flow rate, coolingwater temperatures, water bacterial level, vibrations, seismometermeasurements, strain measurements, temperature, humidity, precipitation,wind speed, wind force, wind direction, electrical output, relativedistances, critical point spatial locations, camera feeds as well asnumerous other sensing.

Referring now to FIG. 3, in some embodiments, aspects of locationdetermination and verification may be integrated into other securitymechanisms that may include processes that involve security specificprotocols and hardware which may involve communications with cloudstorage 304. For example, one or more of the wireless communicationsdescribed herein may also involve the use of a security processor, suchas a chip to cloud security processor 306 and associated software andprocesses. According to the present invention, the chip to cloudsecurity processor 306 and one or more additional apparatus involvingchip to cloud technology may interact with a location coordinategenerator 307 to further ascertain that a requested process involvingdigital content is being requested by an authorized Agent, from anauthorized geospatial area and for an authorized purpose.

In some embodiments, an Agent 300 may position a smart device 305including one or more transceivers and antenna arrays in a firstposition 301 proximate to a portion in a space of interest 310. Thefirst position 301 of the Smart Device 305 may be determined andrecorded, for example in terms of cartesian coordinates X, Y and Z,representative of three planes 302X, 302Y, 302Z. The Agent 300 mayorient the smart device 305 in a general direction of the portion inspace of interest 310. Positional coordinates may also include otherknown coordinate values, such as: polar coordinates and/or cylindricalcoordinates.

A controller in an external system or in the smart device 305 maygenerate a directional indicator 303 (e.g., one or both of a ray and avector). The directional indicator 303 may be directed towards a portionof a space of interest in which the Agent 300 would like to execute asecure transaction, such as interaction with a Virtual Tag 311 (e.g.:generation of a new virtual tag; modification an existing virtual tag;or receipt of digital content included in an existing virtual tag).Interaction with a Virtual Tag 311 may include one or more of: receiptof digital content associated with the Virtual Tag 311; associatingdigital content with the Virtual Tag 311; and conditions for access ofthe Virtual Tag 311.

In some embodiments, the vector may have a length determined by acontroller that is based upon a distance to a feature of interest inspace as represented in a model on the controller in the direction ofthe generated vector. The vector may represent a distance from thesecond position to the space of interest 310 along the axis defined by aline between the first position 301 and the second position. Incontrast, a ray may include just a starting point and a direction.

In still other embodiments, a device with a controller and anaccelerometer, such as mobile phone, tablet or other Smart Device 305that includes a Transceiver, may include a user display that allows adirection to be indicated by movement of the device from a determinedlocation acting as a base position towards a point in a direction ofinterest or representing a center of an RTA of the device. The movementmay occur to a second location in an extended position. In someimplementations, the Smart Device 305 determines a first position 301based upon triangulation with the reference points. The process ofdetermination of a position based upon triangulation with the referencepoints may be accomplished, for example via executable softwareinteracting with a controller in the Smart Device 305, such as, forexample by running an app on the Smart Device 305.

An array of antennas positioned at a user reference point may allow forthe accurate receipt of orientation information from a transmitter. Asdiscussed earlier a combination devices with arrays of antennas may beused to calculation a position. A single Node with an array of antennascan be used for orienteering and determining a direction of interest.Each of the antennas in such an array receiving a signal from a sourcemay have different phase aspects of the received signal at the antennasdue to different distances that the emitted signal passes through. Thephase differences can be turned into a computed angle that the sourcemakes with the antenna array.

In some embodiments, aspects of location determination and verificationmay be integrated with other security mechanisms 306-309 that mayimplement processes that involve security specific protocols andhardware. For example, any of the wireless communications may alsoinvolve the use of a security processor, such as a cloud securityprocessor 306 and associated software and processes. According to thepresent invention, the security processor 306 and other chip to cloudtechnology 308 may interact with a location coordinate generator 307 tofurther ascertain that a requested process involving digital content isbeing requested by an authorized Agent 300, from an authorizedgeospatial area and for an authorized purpose.

In some embodiments, methods, and devices for determining a directionthat may be referenced for one or both of data capture and datapresentation of a particular portion of the virtual representation ofsurroundings of a user. An Agent 300 may position a Smart Device 305 ina first position 301 proximate to a portion in a space of interest 310.The first position 301 of the Smart Device 305 may be determined andrecorded. The Agent 300 may then relocate the Smart Device 305 to asecond position in a general direction of the portion in space ofinterest. With associated position information obtained for the firstand second positions a controller in an external system or in anassociated Smart Device 305 may generate one or both of a ray and avector towards the portion of a space of interest.

In some embodiments, the vector may have a length determined by acontroller that is based upon a distance to a feature of interest inspace as represented in a model on the controller in the direction ofthe generated vector. The vector may represent a distance from thesecond position to the space of interest 310 along the axis defined by aline between the first position 301 and the second position. Incontrast, a ray may include just a starting point and a direction. Invarious embodiments, a vector may be referenced to define a volume ofspace that has a perimeter of a defined shape, such as a rectangle,circle, oval, square or other perimeter shape with a volume defined bythe perimeter being extended through space in a direction and for adistance indicated by the vector.

In still other embodiments, a device with a controller and anaccelerometer, such as mobile phone, tablet or other Smart Device 305that includes a Transceiver, may include a user display that allows adirection to be indicated by movement of the device from a determinedlocation acting as a base position towards a point in a direction ofinterest or representing a center of an RTA of the device. The movementmay occur to a second location in an extended position. In someimplementations, the Smart Device determines a first position 301 basedupon triangulation with the reference points. The process ofdetermination of a position based upon triangulation with the referencepoints may be accomplished, for example via executable softwareinteracting with a controller in the Smart Device, such as, for exampleby running an app on the Smart Device 305.

An array of antennas positioned at a user reference point may allow forthe accurate receipt of orientation information from a transmitter. Asdiscussed earlier a combination devices with arrays of antennas may beused to calculation a position. A single Node with an array of antennascan be used for orienteering and determining a direction of interest.Each of the antennas in such an array receiving a signal from a sourcemay have different phase aspects of the received signal at the antennasdue to different distances that the emitted signal passes through. Thephase differences can be turned into a computed angle that the sourcemakes with the antenna array.

As illustrated in FIG. 3A, in some embodiments, a sonic Transceiver 310Amay transmit a sonic transmission 312A and determine a location 311Abased upon receiving an echo 313A back from an Agent-supported device314A. Walls 315A may also generate reflected sonic emanations 316A.

In some examples, as may be used in orienteering herein, anAgent-supported device 314A may support receivers, transmitters ortransceivers that interact with ultrasonic transceivers fixedly securedto a reference point position, such as via mechanical mounting within aroom environment. An ultrasonic positioning system may have indoorpositioning accuracy at centimeter, millimeter, and even sub-millimeteraccuracy. Multiple ultrasonic Transceivers may transceive from anAgent-supported device to communicate with fixed reference pointtransceivers may transmit signals. Arrival of the sound transmissionsmay be accurately timed and converted to distances. In some embodiments,distance determinations may be improved with knowledge of temperaturesin the environment containing the sound transceiving. For example,temperature may be measured at one or more of: an Agent-supported SmartDevice, a Reference Point position, and an ambient environment.

In some examples, synced timing apparatus is able to generate a locationof a stationary Agent to within centimeter accuracy using sonic wavetransmissions and reception and preferably within several millimeters ofaccuracy. In addition, in some embodiments sensors are able to detectfrequency shifts within the sonic emanations which may add informationabout the relative rate of movement of the Agent, which may then in turnallow for correction to the timing signals.

In some examples, a combination of radio frequency emissions andultrasonic emissions may be used. For example, a complement of radiofrequency emissions/receptions and ultrasonic of radio frequencyemissions and ultrasonic emissions may be reconciled to generate moreaccurate location determination. In another aspect, a radio frequencysignal may be used to transmit syncing signals that establish a timethat ultrasonic signals are transmitted. Since, the electromagnetictransmissions may be orders of magnitude faster than soundtransmissions, the electromagnetic transmissions relatively small timeof travel from the Transceivers to the Agent may be negligible andtherefore used as “zero-time” setpoints as received at theAgent-supported Transceiver. In such embodiments, a controllerdetermining a location may use not only relative arrival times, but alsoa delta time between a radiofrequency transmission and ultrasonictransmission to determine a distance from a transmitting Transceiver. Anarray of such ultrasonic and/or radiofrequency transceivers provideincreased accuracy in triangulating a location of the Agent.

In still further examples, RF communications may not only transmit asyncing pulse, but also transmit digital data about various aspects of adefined area, such as the defined area's identification, its relativeand/or absolute location in space and other refinements. In someexamples, data related to improved distance calculation may also betransmitted by RF communication such as temperature of the environment,humidity and the like which may influence the speed of sound in theenvironment as a non-limiting example. In some examples, such a systemmay result in millimeter level accuracy of position determination.

In some examples, the process may be iterated to refine the direction ofeach of the ultrasonic transmitters and maximize signal levels of thetransmission which may provide additional information in the calculationof a position. RF and Wi-Fi transmissions may be used for datacommunications and syncing as have been described. In other examples, asan Agent-supported device 314A is moving, an iterative process may beused to track the Agent-supported device 314A moves through space.Stationary Agents may be tracked with submillimeter accuracy in someembodiments.

A direction dimension may be based upon multiple transceivers includedin a Smart Device or a Smart Receptacle or via a movement of a SmartDevice or Smart Receptacle while an Agent supporting the Smart Device orSmart Receptacle remains in a stationary position in relation toreference, such as a ground plane position. For example, a device with acontroller and an accelerometer, such as mobile Smart Device, mayinclude a user display that allows a direction to be indicated bymovement of the device from a determined location acting as a baseposition towards a feature in the intended direction where the movementresults in an extended position. In some implementations, the SmartDevice may first determine a first position based upon triangulationwith the reference points and a second position (extended position) alsobased upon triangulation with the reference points.

As described above, facing a mobile device towards an area in aStructure and movement of the mobile device in a particular pattern maybe used to ascertain a specific area in space to be interpreted byvarious methods. In some examples, the leading edge of a smart device,or the top portion of the user screen (in either portrait or landscapemode of display) may be the reference for the direction pointed in bythe user. If the smart device is held at an angle relative to theground, in some examples, the angle formed by the perpendicular to thetop portion of the user screen may be projected upon the ground and thatprojection taken as the indication of direction.

Referring to FIGS. 4A-D a series of exemplary devices employing matrices(or arrays) of antennas for use with Nodes that communicate wirelessly,such as via exemplary UWB, Sonic, Bluetooth, a Wi-Fi, or other modality,is illustrated. Linear antenna arrays 401 are illustrated in FIG. 4A.Rectangular antenna arrays 402 are illustrated in FIG. 4B. Circularantenna arrays 403 are illustrated in FIG. 4C, other shapes for arraysare within the scope of the invention. In addition, an antenna array maybe omni-directional or directional.

In some embodiments, see FIG. 4D item 404, a Smart Device 405 mayinclude one or more Nodes 406 internal to the Smart Device 405 orfixedly attached or removably attached to the Smart Device 405. EachNode 406 may include antenna 407 arrays combined with a power source andcircuitry to form complete self-contained devices. The Nodes 406 or acontroller may determine an RTT, time of arrival, AoA and/or AoD orother related angular determinations based upon values for variablesinvolved in wireless communications. For example, a composite device maybe formed when a Node 406 with a configuration of antennas, such as theillustrated exemplary circular configuration of antennas 407, isattached to a Smart Device 405. The Node 406 attached to the SmartDevice 405 may communicate information from and to the Smart Device 405including calculated results received from or about another Node 406,such as a Node 406 fixed as a Reference Point Transceiver or a Node withdynamic locations, wherein the wireless communications are conducive togeneration of data useful for determination of a position (e.g., timingdata, angles of departure and/or arrival, amplitude, strength, phasechange, etc.). A combination of angles from multiple fixed referencepoints to the antenna array can allow for a location of a user in space.However, with even a single wireless source able to communicate with theantenna array, it may be possible to determine a direction of interestor a device related field of view.

An array of antennas positioned at a reference point may allow for theaccurate receipt of orientation information from a transmitter. Asdiscussed earlier a combination devices with arrays of antennas may beused to calculation a position. A single Node with an array of antennascan be used for orienteering and determining a direction of interest.Each of the antennas in such an array receiving a signal from a sourcemay have different phase aspects of the received signal at the antennasdue to different distances that the emitted signal passes through. Thephase differences can be turned into a computed angle that the sourcemakes with the antenna array.

Referring now to FIG. 5, in some embodiments, one or both of a SmartDevice 501 and a Smart Receptacle 502 may incorporate multipleTransceivers 503-510 and a direction of interest may be ascertained bygenerating a vector 526 passing through a respective position of each ofat least two of the transceivers (as illustrated through transceiver 505and 507). The respective positions of each of the transceivers 503-510supported by the Smart Device 501 and/or Smart Receptacle 502 may beascertained according to the methods presented herein, including forexample via triangulation, trilateration, signal strength analysis, RTT,AoD, AoA, topography recognition, and the like. Reference PositionTransceivers 511-514 may be fixed in a certain location.

In some embodiments, one or both of the Smart Device 501 and the SmartReceptacle 502 incorporating Transceivers 503-510 may be rotated in amanner (such as, for example in a clockwise or counterclockwise movement520, 522 relative to a display screen) that repositions one or moreTransceivers 503-510 from a first position to a second position. Avector 526 may be generated at an angle that is zero degrees 524 with aplane of a display 515 or perpendicular 525 or some other designatedangle in relation to the smart device 501 and an associated displayscreen 523. In some embodiments, an angle in relation to the smartdevice is perpendicular 525 to a display screen 523 and thereby viewablevia a forward-looking sensor (or other CCD or LIDAR device) on the smartdevice. In addition, a mirror or other angle-altering device may be usedin conjunction with a CCD, LIDAR or other energy receiving device.

Movements of a Smart Device 501 equipped with an antenna array can bedetermined via relative positions of the antenna and/or via operation ofan accelerometer (?) within the Smart Device 501 or Smart Receptacle502. Rough movement sense may be inferred with a single source to theantenna array. However, with multiple sources, the positional movementof each of the antennas can be used to sense many types of movementsincluding translations and rotations.

A user may position the smart device 501 such that an object in adirection of interest is within in the CCD view. The smart device maythen be moved to reposition one or more of the transceivers 503-510 froma first position to a second position and thereby capture the directionof interest via a generation of a vector in the direction of interest.

In addition to movement of the Smart Device 501 and/or the SmartReceptacle 502 may include a magnetic force detection device, such as amagnetometer. A registration of a magnetic force may be determined inrelation to a particular direction of interest and a subsequentdetermination of magnetic force referenced or provide a subsequentorientation of the Smart Device 501 or Smart Receptable 502.

In some embodiments, the magnetic force detection device may be usedcombination with, or in place of directional movement of the SmartDevice transceivers 503-510 to quantify a direction of interest to auser. Embodiments may include an electronic and/or magnetic sensor toindicate a direction of interest when a Smart Device 501 and/or SmartReceptacle 502 is aligned in a direction of interest. Alignment mayinclude, for example, pointing a specified side of a Smart Device 501and/or Smart Receptacle 502, or pointing an arrow or other symboldisplayed upon a user interface on the Smart Device 501 towards adirection of interest.

A magnetic force detection device (sometimes referred to as a magneticsensor) may detect a magnetic field particular to a setting that a SmartDevice is located. For example, in some embodiments, a particularstructure or other setting may have a magnetic force that is primarilysubject to the earth's magnetic field or may be primarily subject toelectromagnetic forces from equipment, power lines, or some othermagnetic influence or disturbance. An initial quantification of amagnetic influence at a first instance in time may be completed and maybe compared to a subsequent quantification of magnetic influence at alater instance in time. In this manner an initial direction of interestindicated by a position, orientation, pitch, and yaw of a Node, such asa Smart Device may be compared to a subsequent position, orientation,pitch, and yaw of the Smart Device.

In some embodiments, an initial position, pitch, and yaw of a SmartDevice 501 may be described as a relative angle to a presiding magneticforce. Examples of presiding magnetic forces include, magneticinfluences of electrical charges, Earth's magnetic field, magnetizedmaterials, permanent magnetic material, strong magnetic fields,ferromagnetism, ferrimagnetism, antiferromagnetism, paramagnetism, anddiamagnetism, or electric fields that are generated at reference nodesat known positions which may be locally used to indicate a direction ofinterest.

Smart devices may include electronic magnetic sensors as part of theirdevice infrastructure. The magnetic sensors may typically includesensing elements deployed along three axis. In some examples, themagnetic sensors may be supplemented with electronic accelerometers,such as MEMS accelerometers.

In some examples the magnetic sensor may include a Hall effect sensorthat is operative to measure a sensed magnetic field perpendicular tothe body of the sensor via operation of the Hall effect sensor. In someexamples, a Hall effect sensor may be built into silicon to generate arelatively sensitive sensing capability for magnetic fields. In someHall effect sensors, electrons and holes flowing in a region of thesilicon may interact with the regional magnetic field and build up onthe fringes of the conduction region, thus generating a measurablevoltage potential. In other examples, anisotropic magnetoresistancesensors may sensitively detect the magnetic field at the device as asignificant change in resistance of structure elements in the device.

In still further examples, the magnetic sensor may include one or moregiant magnetoresistance (GMR) sensors may detect the magnetic field. Insome of these examples, the GMR sensors may detect a magnetic field witha current-perpendicular-to-plane (CPP) GMR configuration. In otherexamples, a current-in-plane (CIP) GMR sensor configuration may be used.The resulting three-axis magnetic sensors may perform a sensitivecompass function to determine a direction of a specified portion of theSmart Device and/or an edge of the smart device relative to the localmagnetic field environment. A specified portion of the Smart Device maybe indicated via a user interface presented on a screen of the SmartDevice.

Referring now to FIG. 6, as illustrated, a vector in a direction ofinterest 625 may be based upon a forward arcuate movement 623 or acomplementary arcuate movement 624 of the smart device 601, such as amovement of an upper edge 618 in a forward arcuate movement 623. Thelower edge 619 may also be moved in a complementary arcuate movement 624or remain stationary. The movement of one or both the upper edge 618-and lower edge 619 also results in movement of one or more transceivers503-510 (as shown in FIG. 5) and/or registration in an onboardaccelerometer 634. The movement of the transceivers 503-510 (againillustrated in FIG. 5) may preferably be a sufficient distance toregister disparate geospatial positions based upon wirelesstransmissions and/or sufficient to register movement via theaccelerometer 634.

As presented herein, a direction dimension may be based upon one or moreof: a wireless position of two or more transceivers, a movement of adevice, a magnetic force determination, a LIDAR transmission andreceiving, CCD energy determinations and other assessments of anenvironment containing the Smart Device and/or Smart Receptacle. Forexample, a device with a controller and an accelerometer, such as amobile Smart Device, may include a user display that allows a directionto be indicated by movement of the device from a determined locationacting as a base position towards a feature in the intended directionwhere the movement results in an extended position. In someimplementations, the Smart Device may first determine a first positionbased upon triangulation with the reference points and a second position(extended position) also based upon triangulation with the referencepoints.

As described above, facing a mobile device towards an area in aStructure and movement of the mobile device in a particular pattern maybe used to ascertain a specific area in space to be interpreted byvarious methods.

Referring to FIG. 7, an illustration of an Agent 750 utilizing anoriented stereoscopic sensor system 751 and 754 to orient a direction ofinterest is shown. The stereoscopic sensor systems 751 and 754 mayobtain two different images from different viewpoints 752-753 which maybe used to create topographical shape profiles algorithmically. Acontroller may obtain the image and topographic data and algorithmicallycompare them to previously stored images and topographic data in thegeneral environment of the user. The resulting comparison of the imageryand topography may determine an orientation in space of the Agent 750and in some examples determine a device field of view. The controllermay utilize this determined field of view for various functionality asdescribed herein.

Referring now to FIG. 8A, a side view illustrates a WCA 800 surroundinga Node, such as a Smart Device 802. Energy 803, which is illustrated asrays, is received by one or more energy-receiving Sensors 804 in theSmart Device 802 (energy-receiving Sensors may also be in a SmartReceptacle associated with the Smart Device, though this is notillustrated in FIG. 8A). An exemplary ray may proceed from a position805 within RTA 801 boundary to the energy-receiving Sensor 804.

As illustrated, a portion of the RTA 801 may flatten out 801 a inresponse to a ground plane, wall, partition, or other obstructionencountered. A Node 806 may be located on or within a surface that makesup a relevant obstruction and the Node 806 may appear to be along aperimeter of the RTA 801. Similarly, a Virtual Tag may be associatedwith location coordinates that appear on or within a floor, wall,partition, or other article acting as a radio frequency obstruction andthereby appear to be a part of the obstruction, however, since it isvirtual, the Virtual Tag will not affect the physical properties of theobstruction. Essentially, a Virtual Tag may have location coordinatesthat correspond to anywhere in the physical real-world. In someexamples, a software limit or setting may limit location coordinates ofVirtual Tags to some distance from a base position or a distance from adesignated position, such as a location of a designated Physical Tag,Reference Point Transceiver, or other definable position.

In addition to obstructions, a topography of an environment within anRTA 801 may also limit wireless conveyance of energy within an RTA 801to an energy-receiving Sensor 804. Topography artifacts may include, forexample, a terrain, buildings, infrastructure, machinery, shelving, orother items and/or other structures that may create impediments to thereceipt of wireless energy.

Energy 803 received into the energy-receiving Sensor 804 may be used tocreate aspects of a user interface that is descriptive of theenvironment within the RTA 801. According to the present invention,environmental aspects, Nodes 806, Tags (both physical Tags and VirtualTags) and the like may be combined with user interactive mechanisms,such as switches or other control devices built into a user interactivedevice and included in a user interactive interface. For example, energylevels received into an energy-receiving Sensor 804 may be combined withlocation coordinates of Physical Tags and/or Virtual Tags and a userinteractive device may be positioned in an interactive user interface ata position correlating with the position coordinates and be surroundedwith a visual indicator or the received energy levels.

In this manner, a single user interface will include a static imagerepresentative of received energy levels at an instance in time; avisual representation of a location(s) of Physical and/or VirtualTag(s), and devices with user interactive functionality. In someembodiments, the devices with user interactive functionality may bepositioned at a location in the user interactive interface correlatingwith the position(s) of the Physical and/or Virtual Tag(s).

This disclosure will discuss RTAs 801 that are frustums of a generallyconical shape, however, RTAs 801 of other volume shapes are within thescope of the invention. For example, if an energy-receiving Sensor 804included a receiving surface that was a shape other than round, or hadmultiple receiving surfaces, each of a round or other shape, the RTA 801associated with such an energy-receiving Sensor may have a shape otherthan a frustum of generally conical shape.

Referring now to FIG. 8B, a top-down view of an RTA 801 is depicted. AnRTA 801 will include some portion of a WCA 800. As illustrated, the WCA800 includes a space with irregular boundaries encompassing 360 degreesaround the Smart Device 802. Aspects such as topography, strength ofsignals and atmospheric conditions (or other medium through which awireless communication will travel) may affect and/or limit a perimeterof the WCA 800. A location of the RTA 801 may be referenced to determinewhich Tags (Physical and/or Virtual) such as Node 806 are includedwithin an interactive user interface. Generally, preferred embodimentsmay only include Tags with location coordinates with the RTA 801 in theinteractive user interface. However, embodiments may include Tagsexternal to the RTA 801 in a particular interactive user interface.

Referring now to FIG. 8C, a side view of a WCA 800 is presented where anenergy-receiving Sensor 804 is capable of quantifying a particular formof energy, such as a particular bandwidth of energy received from a userselected RTA 807. A Smart Device 802 may incorporate or be in logicalcommunication with multiple energy-receiving Sensors 804, eachenergy-receiving Sensor capable of quantifying a limited energy spectrumin an environment defined by the RTA 807 selected by the user.

Some embodiments include an RTA 807 that varies according to a type ofenergy-receiving Sensor 804 receiving a corresponding type of energy.For example, an energy-receiving Sensor 804 that receives energy in alower bandwidth may have an RTA 807 that extends a greater distance thanan energy-receiving Sensor 804 that receives energy in a higherbandwidth. Similarly, some energy-receiving Sensors 804 may be affectedby forces outside of the RTA 807, such as a magnetometer which may besensitive to signal interactions around all of the WCA 800, and an RTA807 associated with a magnetometer may accordingly be the same as theWCA 800.

By way of non-limiting example, an RTA 807 for a CCD-type energyreceiver may be represented as a frustum with an expansion angle ofapproximately 60 degrees in shape. Accordingly, the RTA 807 subtendsonly a portion of the universal WCA 800.

Referring now to FIG. 8D, a top view of a WCA 800D is illustrated withan RTA 807A comprising a frustum with an expansion angle ofapproximately 60 degrees. A Smart Device 802 with an energy receiverthat quantifies a specified bandwidth of energy from the RTA 807A maygenerate a user interface with an image based upon energy quantifiedfrom RTA 807A.

In FIG. 8D, the WCA 800D is represented as a spherical area. A WCA 800Dmay be designated that is less than an entire area of possible radiocommunication using a specific designated wireless communicationmodality. For example, WCA 800D may be spherical and stay withinboundaries of a modality based upon a UWB wireless communicationprotocol.

A user interface based upon quantified energy in an RTA 807A, maypresent a representation of energy within the respective RTA 807A asquantified by an energy-receiving Sensor in a Smart Device 802. Energylevels of other three-dimensional spaces within the WCA 800D may bequantified by energy receivers and presented in a user interface bydirecting energy from a selected three-dimensional space into the energyreceivers and thereby defining a different RTA 807A. In this manner,energy levels may be quantified from essentially any area within the WCA800D and represented as part of a user interface. Quantified energylevels may vary based upon a receiving Sensor. For example, a CCD Sensormay quantify visible light spectrum energy, and a LIDAR receiver a broadspectrum, an infrared receiver may quantify infrared energy levels, andenergy-receiving Sensors. A particular feature present in a particularportion of the electromagnetic spectrum quantified by anenergy-receiving Sensor may have a unique physical shape whichcharacterizes it, and which may be associated with a correspondingvirtual-world aspect and Tag associated with the location.

In some examples, as has been described, quantification of energy levelsassociated with aspects of the physical world may be for one or more of:characterizing an RTA 807A by quantifying energy levels and patternsexisting at an instance in time; determining a location and/ororientation of a Smart Device 802 or other Node, such as Node 806; andverifying a location and/or orientation of a Smart Device 802. Variousembodiments include energy levels associated with aspects of thephysical world may be communicated by the Smart Device 802 to a remotecontroller for further processing, and the remote controller maycommunicate information back to the Smart Device or to another userinterface. Information communicated from the controller may include, forexample: an orientation of physical and/or virtual aspects locatedwithin the universal RTA in relation to the Smart Device; and quantifiedenergy indicating of or more of a topographical feature, a surfacetemperature, a vibration level, information associated with a VirtualTag, information associated with a physical Tag, sensor data, or otherinformation associated with the RTA 807A.

A view of an RTA 807A (Radio Target Area) may be a relatively smallportion of the entire wireless communication area (WCA) that surrounds aSmart Device. An area of energy to be quantified by a sensor (sometimesreferred to herein as the Radio Target Area) may be displayed surroundedby the WCA 800D.

Referring now to FIG. 8E, an exemplary presentation of an RTA 812superimposed upon a representation of a WCA 811 is illustrated. The WCA811 is illustrated with a perspective view of a spheroid with analignment feature 814 such as a spheroid dividing arc, or a line. Ablackened ellipsoid feature is a representation of the RTA 812associated with a particular Smart Device which would be located at acenter of the spheroid WCA 811. If desired, one or more energy receivingdevices associated with or incorporated into a Smart Device may berepositioned or have a changed orientation in space to ultimately scanall of the accessible universal Radio Target Area space.

Referring to FIG. 8F, an illustration of how moving the one or moreenergy receiving devices around in space may alter an area defined asthe RTA 813. The same orientation of the universal WCA 811 may be notedby a same location of the alignment feature 814. Relative movement ofthe ellipsoid feature illustrates a change in an area designated as RTA813.

Referring to FIG. 8G, an illustration of adding Tag locations (which maybe Physical Tags or Virtual Tags) to a mapping of the WCA 811 isprovided. A Tag may be represented in the WCA, for example, as an icon(two- or three-dimensional) positioned in space according to acoordinate system, such as Cartesian coordinates, polar coordinates,spherical coordinates, or other mechanism for designating a position.Coordinates may specify one or both of physical real-world Tags andVirtual Tags.

A location of a real-world Tag or Virtual Tag may be in either RTA 813,the WCA 811 or external to both the RTA 813 and the WCA 811. Examples ofTags outside the RTA 813 and within the WCA 811 include Tags 815-819. Anexample of a Tag in the device RTA is Tag 820. A Tag located external toof the WCA 811, and the RTA 813 includes Tag 821.

In some examples, a display on the user's Smart Device may illustrateimage data captured via a CCD included in a Smart Device. Portions ofthe image data captured via a CCD may be removed and replaced with anicon at position correlating to a position in space within the RTA 813.The icon may indicate of a Tag 821 located within the RTA 813, or atleast the direction in the RTA 813 along which the Tag 821 may belocated at an instance in time. In addition, an area of a user interfaceportraying the Icon may user interactive device such that when thedevice is activated, the Smart Device is operative to perform an action.

The actual positions of the Tags in real-world space (or the digitalequivalent in the real-world space) may be stored and maintained in adatabase. Positions of physical Tags may be determined via techniquesbased upon wireless communication and be updated periodically. A periodof update may be contingent upon variables including, user preference,Tag movement, change in environmental conditions, User query or othervariable that may be converted into a programmable command. In anotherexample of some embodiment, an Agent may interact with a user interfaceand understand the presence of Tags that are outside of the RTA 813 andadjust one or both of a position and direction that the Smart Device tocause the Smart Device to be positioned such that the RTA 813encompasses a position of the Tag of interest.

Referring to illustration FIG. 9A, an exemplary apparatus foreffectuating the methods described herein is shown, wherein Smart Device901 has within its Radio Target Area (RTA 905) an infrastructure 906.Smart Device 901 may display a user interface on a touchscreen 902 basedupon data generated by an energy-receiving Sensor 903 incorporated intothe Smart Device 901 or operative in conjunction with the Smart Device901. The energy-receiving Sensor 903 may produce data representative ofan area from which the energy-receiving Sensor 903 received energy. Auser interface on a touchscreen 902 may be generated that is based uponrelative values of some or all of values for data variables produced bythe energy-receiving Sensor 903.

Smart Device 901 may have its position and direction of orientationdetermined using the orienteering methods described herein, withreference to one or more Reference Point Transceivers 908 A-D. Theposition may be determined relative to a Base Node, such as ReferencePoint Transceiver 908A. The Base Node may operate as an origin in acoordinate system associated with infrastructure 906 and itssurroundings. A position-determination process may be aided withreference to transmitter 907, which in some embodiments, may be aReference Point Transceiver. In this example, transmitter 907 ispositioned proximate to the infrastructure 906.

A receiver on Smart Device 901 may be operative to receive a wirelesslogical communication from transmitter 907. This communication may be inone of a variety of modalities, such as Bluetooth, ultra-wideband (UWB),radiofrequency, infrared, ultrasound, etc. Based upon the signal, SmartDevice 901 may transmit a database query based upon a determined set ofcoordinates of transmitter 907, a set of coordinates of the Smart Device901, the RTA 905, or other position and direction relevant variable.

If the database contains an entry comprising a set of coordinates (as adata structure) and the set of coordinates define a point withindisplayable distance to the set of coordinates of transmitter 907, thenthe user interface on a touchscreen 902 may display an interface thatincludes an interactive area 904 that manifests a Virtual Tag (such asan icon or other definable area) in context to a virtual infrastructurerepresentative of the infrastructure 906. In this way, a user of SmartDevice 901 may be alerted to the presence of information associated withinfrastructure 906 in which the user may be interested.

In some embodiments, inclusion of an interactive area 904 on the userinterface on a touchscreen 902 may be contingent upon an Agent operatingthe Smart Device 901 presenting appropriate credentials and/orpermissions to access digital content made accessible via theinteractive area. Still further appropriate credentials and/orpermission may be required to ascertain that an interactive area exists.For example, an image displayed on a user interface may include imagerydescriptive of the infrastructure. A user with proper credentials may bepresented with a user interface that includes an interactive area 904that manifests a Virtual Tag 904A, such as an icon or outline of imagerydescriptive of an item of equipment or a person; while a user who hasnot presented proper credentials may not be made aware of the existenceof such an interactive area, nor the content included in the Virtual Tag904A associated with the interactive area 904.

In another aspect, in some embodiments, an interactive area 904 may onlydisplay if Smart Device 901 is in active communication with a specifiedWi-Fi network, or if the Smart Device 901 his in communication with atleast one of a specified Node or Nodes. Communication with a Node mayinclude, for example, wireless communication via a wireless modality,such as, one or more of: UWB; Bluetooth, infrared, sonic, or othermodality discussed In other embodiments, interactive area 904 maydisplay on any user interface on a touchscreen 902 (if the RTA 905includes transmitter 907), but further functionality may be based uponsuccessfully responding to a security challenge. A security challengemay include, for example, a biometric measurement, inputting a password,correctly input an answer to a question, a gesture made with the SmartDevice, a gesture made in communication with a sensor integrated withinor with a Smart Device (such as, for example, motion of hand(s) in frontof a camera, or motion of a hand wearing a Smart Ring or a wrist wearinga Smart Wristband).

In some embodiments, the appearance of interactive area 904 may changebased upon variables, such as, one or more of: the position of the SmartDevice 901; the identity of user or Agent; if the interactive area 904is related to a query and/or query response; if the interactive area 904is within an RTA 905 or based upon some other dynamic. For example, ifthe user has a certain UUID, and the database includes a messagespecifically intended for a user with that UUID, then the interactivearea 904 or an icon may flash to indicate the presence of a message.This message may be displayed textually, visually, audibly, or by ahologram. Similarly, the database may record one or more instances inwhich the Smart Device 901 is accessed via a query from a Smart Device.Access via a query may be associated with a time stamp. If data relatedto infrastructure 906 has changed since a previous time stamp, theninteractive area 904 may be presented in a highlighting color (such as,for example be presented in red or other color) to indicate that achange has been detected. In addition, digital content may be appendedto any content already in the database, such as additional alphanumericannotation, an audio file, an image file, a video file, or a story file.

In some embodiments, in response to activation of an interactive area904 (such as a click, screen touch, voice command, gesture, etc.),additional functionality may be provided via the Smart Device 901 orother item of equipment. For example, selecting interactive area 904 maydisplay digital content related to infrastructure 906. Alternatively,activating the interactive user device associated with interactive area904 may generate a control panel, which may allow the user to controlaspects relating to sensors or other electronics within infrastructure906. For example, upon confirmation that Smart Device 901 has theappropriate permissions, selecting interactive area 904 (or otheractivation of the interactive area 904) may allow the user to turn offselected lights within infrastructure 906. Still other embodiments allowfor activation of an interactive area 904 to be a prerequisite tooperation of equipment located within the RTA 905 or other defined area.

An interactive area 904, may be incorporated into a user interface inalmost any manner conducive to a user activating the interactive area904. For example, a user interface that recreates a visual of a physicalarea, such as, by way of non-limiting example: an image (or video) basedupon a CCD sensor array; a two dimensional representation of a physicalarea (such as a floorplan, site plan or architectural drawing); and athree dimensional representation (such as a CAD model, AVM; or AugmentedReality model) may include interactive areas that include areas of theimage data, integration of one or more of: an icon, an outlined imagearea, a highlighted image area, an image area with a changed appearance(e.g. change in hue or color), integration or overlay of an image (e.g.a logo, emoticon, or other device).

The Smart Device 901 may also display other functional buttons on itsuser interface on a touchscreen 902. In some examples, one such functionmay be to show displays of the sensor in the context of the universalRTA 905 surrounding the user. By activating the functional button, theuser may be presented with a set of options to display the universal RTA905.

According to the present invention, an interactive area 904 may be usedto retrieve digital content, and/or to store digital content forsubsequent retrieval. Digital content may be associated with one or moresets of position coordinates (e.g., cartesian coordinates, polarcoordinates, and/or cylindrical coordinates). A user interface on atouchscreen 902, AVM and/or two dimensional representation of aninfrastructure or geospatial area may be produced that allows thedigital content to be accessed based upon the associated coordinates.The retrieval of the digital content is persistent in the sense that itmay be retrieved, and new digital content may be added for so long as anunderlying infrastructure enabling determination of coordinates used toaccess and/or place digital content within a coordinate frameworkexists. Referring to FIG. 9B, nonlimiting and exemplary apparatus andmethods for presenting a user interface including an RTA is illustrated.The display screen of the Smart Device 901 may display a number ofinformational digital content components. In some embodiments, a similarillustration as FIG. 8G may be included as an inset 923 of the userinterface 920. In addition, in some embodiments, a user interface 920may include a representation of the RTA may be formed by flattening thesurface of the illustrated sphere 925 into a flat depiction with surfaceregions flattened into a segment 921. The RTA may be illustrated on theflat segments. A user interactive 920 (which may be an icon, highlightedarea, outline, portion of an image, or other defined area) may belocated within the user interface representing the RTA andinfrastructure. The interactive area may also be included in the realtime display of a representation of data generated by anenergy-receiving Sensor 922. Tags may be located within or outside ofthe RTA such that an Agent may move the Smart Device 901 to redirect anRTA to align the RTA into a position that encompasses Tag 924 the Agentwishes to include.

Referring to FIG. 9C, a nonlimiting exemplary user interface 937generated on a display screen 930 for Smart Device 901 is illustrated.The user interface 937 may be displayed, for example, when a userselects an interactive area associated with a Tag 931-932.

The Tag 931-932 may be located at a physical location within or outsideof the RTA 932 (is illustrated). Selection of the Tag (sometimesreferred to activating the Tag 931-932), a menu 933 may display. Amongstthe various information such as text, imagery, video content and thelike that may be displayed an identification of the Tag 934, associatedtextual information and data 935 as well as functional buttons 936 maybe displayed on the user interface and may be used by the user toactivate additional function including new display layers, contentintegration and control function such as in a non-limiting sense acontrol to revert to a previous menu display.

In some examples, a Smart Device may function as a Tag. The Tagfunctionality may include providing location-related information asbroadcasted digital content. In providing such broadcasted digitalcontent, the Smart Device tab may employ numerous forms of securityprotocols for the protection of the information and authorization of itsuse which may include sign-in/password protocols, sharing of encryptionkeys and the like. In similar methods, a central server may providecontent related to a Tag and may manage security protocols and the likewhere a Smart Device acting as a Tag may merely share an identificationphrase that a user could use with applications running or connectingwith the central server could use to be authorized for additionalcontent. Location may be determined by the various means as describedherein including wireless communication with position Nodes by GPS,Cellular, Wi-Fi, Ultrawideband, Bluetooth and the like. If the SmartDevice is operating in a mesh Node, the mesh could communicate withinNodes relative and absolute location information which the Smart Devicemay share as its role as a Tag. In addition to location, other sensordata at the Smart Device such as temperature, vibration, sensor imagery,LiDAR scan imagery, sound sensing.

In addition to real-world data, the Smart Device Tag may also providevirtual content associated with itself and its connected environment.The Smart Device may provide content stored within its memory devicesand may provide dynamically calculated results of processing on contentstored in its memory devices. The virtual content may also correspond toa user interface of the Smart Device Tag that may be used to initiate orauthorize function of the Smart Device including real-world activitiessuch a communication via internet protocol, text, phone, or video.

In some embodiments, an energy-receiving Sensor may receive energyassociated with a LiDAR transmission and/or other functionality involvedin LiDAR scanning which can be used to interrogate the local environmentfor physical shapes. In a Smart Device Tag function, the Smart Devicemay stream its video and scanning data directly or through a servermodel. Some Smart Devices may be configured to operate as a smartsecurity monitoring systems and may provide the video, topographic,audio, and other sensor streams as Tag related content. There may benumerous manners that a Smart Device could function as a Tag in anenvironment.

A Smart Device with either a single- or multiple-sensor system may alsohave a LiDAR scanning capability or other three-dimensional scanningcapability. The Smart Device may utilize a number of systems to refineand improve its accuracy in determining the location that it is at. Inan example, a Smart Device may utilize a GPS or cellular system to getan approximate location of the device. In a next step, a user mayinitiate the Smart Device to take a series of image and scanning dataacquisitions of its environment. For example, the user may move thephone by hand to different directions while maintaining their feet in afixed location. The phone may use one of the orientation methods as havebeen discussed to determine its orientation as it is moved to differentvantage points. The Smart Device may either process those images andcompare against a database in its memory, or it may communicate the datato a server to do the comparison. With an approximate location, theorientation information, and the streams of video and/or topographicinformation, a calculation may be performed to match theimage/topographic information to a more exact positional location. Inalternative examples, the device may use the image and/or topographicinformation to determine the orientation of the device itself.

In some examples, the Smart Device may act as a receiver of one ormultiple types of wireless energy input. For example, the acquisition ofdata based upon a visual light spectrum (approximately 380 to 700 nmwavelength) may be modelled as spatially-characterized electromagneticenergy. Electromagnetic energy in the visible band may enter a focusinglens and be focused up an array of devices. The devices may beCMOS-active pixel sensors, CMOS back-illuminated sensors, or CCDs, asnon-limiting examples, to receive the energy and convert it intospatially-arrayed pixel data.

In some examples, the Smart Device may have an energy-receiving Sensorincorporated or attached which may quantify energy levels forfrequencies outside the visible spectrum. Any optics employed in suchsensors may be different from the previously discussed CMOS and CCDSensors since some of those energy receiving devices may have filters orlenses that absorb wavelengths outside of the visible spectrum. Sensorswith infrared capabilities may have specialized optics and may usedifferent materials for the CMOS and CCD elements—such as indium galliumarsenide-based sensors for wavelengths in the regime of 0.7-2.5 μm.

Alternatively, entirely different sensing elements, such as bolometers,which sense temperature differences of the incoming radiation, may beemployed for longer wavelengths in the regime of 7-14 μm and may includefilters that remove other wavelengths. A display of an infrared Sensor,which senses incoming energy in the infrared band, may be rendered on atypical visual display, but the colors of such displays may have nodirect physical meaning. Instead, a color scheme may be instituted torepresent different infrared wavelengths with different visible colors.Alternatively, the colors may be used to represent different intensitiesof infrared energy received across bands of infrared wavelengths.

In some examples, a Smart Device may both project and receive energy.For example, a Smart Device may scan the topography of its surroundingsby use of LiDAR. In LiDAR, a laser may be used to emit energy into theenvironment. The energy may be emitted as pulses or continuous trains,and the light source may be scanned across the environment. Lightemitted from the Smart Device may proceed into the environment until itis absorbed or reflected. When it is reflected and subsequently receivedat the Sensor, the transit time can be converted to distancemeasurements of the environment. Many different wavelengths of light maybe used to scan an environment, but numerous factors may favor certainchoices such as invisibility to human/animal eyes, safety, absorption bythe airspace surrounding the user and the like. Atmospheric gases mayabsorb significant amounts of infrared transmissions at certainfrequencies; therefore, for LiDAR to be effective in the infraredspectral region, certain bands of emitted frequencies may be favored. Astandard LiDAR system may operate at a band from 900-1100 nm infraredwavelength or at a band centered at approximately 1550 nm. As discussedpreviously, select optic components and materials may be useful forthese wavelengths and the detectors may have improved function based onmaterials such as “black” silicon, germanium, indium phosphide, galliumarsenide, and indium gallium arsenide as exemplary detector materials.

In an example, a laser light source may be rastered across a dimensionof forward looking positions of a Smart Device, which may be representedby a conic section or Radio Target Area in front of the Smart Device. Asthe light is raster across the surface it can address, it may be pulsedon or off. As the light travels out along a collimated path, it mayinteract with a surface and a portion of the intensity may be reflectedbackwards.

A resulting reflected ray may come back to the Smart Device and bereceived by a Sensor in the device. Since the emitted light source maybe orders of magnitude more intense than the surroundings, reflectedlight may dominate a background intensity and the signal detected may becompared with the time of the leading edge of the laser pulse. Therepeated acquisition of the timing signals in the various directions canbe used to form a point cloud that represents the distance to reflectivefeatures from the Smart Device.

As mentioned previously sound may be reflected off of surfaces and thetransit time may be used to characterize a distance between a focusedultrasonic transducer and a reflective surface. In similar manners,points or lines of focused sound emissions may be pulsed at theenvironment and a sensor or array of sensors may detect the reflectedsignals and feed the result to a controller which may calculate pointcloud representation or other or topographic line representations of themeasured surface topography. In some examples, ultrasonic focused andscanned soundwaves in the frequency range of hundreds of megahertz mayresult in small, focused sources whose reflections may be detected bymagnetic or piezoelectric sound transducers as non-limiting examples.

A Smart Device may have numerous different types of energy-collectiondevices which may characterize data values with spatial relevance. Asmentioned before, infrared imaging may be performed on some SmartDevices, and a user may desire to view a spatial representation of theinfrared imaging that represents the data as it may appear if the user'seyes could perceive the energy. In some examples, data values for thewireless energy sensing of infrared energy may be assigned color valuesand displayed in an image format. For examples, low levels of infraredenergy, which may relate to colder temperatures in the imaged regions,may be assigned blue color values, and high levels of infrared energy,which may relate to warmer temperatures, may be assigned red colorvalues. Other color assignments to data values may be used. A legend forthe conversion of the color values to the data values may be provided.

In some examples, the data descriptive of spatially descriptive energylevels quantified by an energy-receiving Sensor data may be portrayed ina user interface. In some user interfaces, representations based uponspatially representative energy levels of different wavelengths may beaggregated or otherwise combined in one or more related user interfaces.Such a combination may allow a user to understand the regional nature ofvarious quantified energy.

In some examples, a user interface may allow for display of thepositional location image points. In some examples, a location of apixel element chosen by a user may be converted to a real-world locationwithin the RTA which may be represented in Cartesian coordinates (X,Y,Z)or in other coordinate systems such as polar coordinate systemsinvolving angles and distances as discussed previously. In someexamples, topographic data obtained by scanning an area with an RTA maybe used quantify topography within the RTA. A user interface based uponsuch quantified energy levels may include virtual presentations of thequantified energy levels from different perspectives and may allow forcoordinate grids (Cartesian or other) to coordinate placement of facetsof a user interface based upon combinations of energy level data, Taglocations and perspective relevance.

In some examples, distinct structures within the RTA may be highlightedand assigned positional coordinates. In some examples, this may occur byimage processing directly, in other examples a user interface may allowfor a user to pick items/regions of interest in an RTA presentation.

In other examples, real and virtual Tags may exist within the RTA. Aphysical Tag may include a position Node, another Smart Device, or anydevice with communication capability that can communicate with either aposition Node or with the Smart Device of the user directly. Suchphysical Tags may be located in numerous manners. In some examples, thephysical Tag may have a direct determination of its location eitherbecause it is stationary and has been programmed with its location orbecause it has the capability of determining its own position with thevarious methods as have been described herein. In other examples, aphysical Tag may be able to communicate with Nodes such as ReferencePoint Transceivers and a location may be determined based upon anexchange of data, such as timing values, in the wireless communications.A Node may also be functional to determine, store and communicate alocation of other Tags. The Smart Device of the user may gain access tothe locations of Tags, either because they are publicly available orbecause the user has established rights digitally to obtain theinformation from some or all of these physical Tags.

There may also be virtual Tags that are associated with positionalcoordinates. The distinction of these Tags over physical Tags is thatthere may be no physical presence to the virtual Tag. It may be adigital or virtual-world entity that has an association with areal-world positional coordinate. Except for this distinction, a virtualTag and a real-world Tag may behave similarly with respect to theirassociation with a physical coordinate.

In these examples, an interactive user interface based upon energylevels and Tags located with an RTA may have icons associated with theplacement of Tags. The user interface may include an icon positionaldesignation and a graphic to indicate the presence of a Tag. It may beapparent that, in some cases, multiple Tags may lay along a singledirection from a given Smart Device location and RTA, and thus multipleicons may be included within a user interface in close proximity. Theuser interface may indicate multiple Tag icons by color changes,blinking or other indicators. As an RTA is changed, Tags along a sameperspective may resolve into different directions for Tags withdifferent positional coordinates.

The Tag icon may indicate to the user a digital functionality associatedwith a real-world or virtual Tag. For example, the icon may allow a userto choose the functionality of the icon by moving a cursor over the iconand making a keystroke or mouse click or for touch screens by pressingthe display location of the Tag icon. The choosing of the Tag icon mayactivate user interface dialogs to allow the user to control subsequentfunctionality. In cases of superimposed Tag icons on a same pixellocation in a user display, a first functionality may allow the user tochoose one of the multiple Tag icons to interact with. In some examples,a Tag icon may be displayed with an associated ID/name and a user mayselect the icon with voice commands rather than physically selecting theicon as described previously. Displays of these Tags may follow similarprotocols as have been discussed in reference to FIGS. 9A-9C.

Referring now to FIG. 10A, a method for generating an augmented-realityRadio Target Area for a Smart Device is shown. At step 1001, wirelessenergy of a first wavelength is received into a wireless receiver. Inexemplary embodiments, this step may include receiving image data basedon visible light into a sensor of the Smart Device. The wireless energymay be dispersed over a one-, two-, or three-dimensional space in adefined physical area, and may be received into a one-, two-, orthree-dimensional array in the receiver. The wireless energy may takethe form of electromagnetic radiation, such as light in thehuman-visible light spectrum (generally having a wavelength between 380nm-740 nm), ultraviolet light (generally having a wavelength between10.0 nm-400 nm), or infrared light (generally having a wavelengthbetween 740 nm-2.00 mm) as examples. The set of wireless energyavailable to the wireless receiver is the Smart Device's Radio TargetArea.

The wireless receiver may be a Smart Device sensor, including a CMOSactive pixel sensor, a CMOS back illuminated sensors, CCD, or a LIDARapparatus, including a solid-state/MEMS-based LIDAR. The wirelessreceiver may comprise an array or other plurality of other wirelessreceivers. The wireless receiver may be operative to receive thewireless energy into an array of an appropriate dimension for subsequentdisplay (possibly after processing) on the Smart Device. For example,where the wireless receiver is a Sensor, the Sensor may be operative totranslate the wireless energy into a two-dimensional array.

At step 1002, a pattern of digital values is generated based uponreceipt of wireless energy into the wireless receiver. This pattern ofdigital values may be based on one or more qualities of the receivedwireless energy, including its intensity, spatial dispersion,wavelength, or angle of arrival. The pattern may be placed into anappropriate array. For example, if the display of the Smart Device is atwo-dimensional display, then the pattern of digital values may comprisea two-dimensional representation of the image data received. In someembodiments, the pattern of digital values may be based on an aggregatedset of values from an array of receivers. For example, if the basis ofthe digital values is the intensity of the wireless energy received intothe receiver, then the digital value assigned to a given entry in thearray may be based on a weighted average of intensity of wireless energyreceived at a plurality of the receivers in the array. Optionally, atstep 1003, the wireless receiver may receive the wireless energy as ananalog signal (for example, if the wireless receiver is ablack-and-white sensor or an unfiltered CCD), and convert the analogsignal to digital values through filtration or other analog-to-digitalconversion. The set of digital values within the Radio Target Area isthe Digital Radio Target Area.

With the Smart Device wireless receiver's Radio Target Area determined,the Smart Device's position should be determined as well, along with thepositions of any items of interest in a given space. Collectively, theSmart Device and the item of interest may comprise wireless Nodes.Accordingly, at step 1004, coordinates representative of a wireless Nodemay be determined relative to a base Node. These coordinates may bedetermined in any appropriate coordinate system (such as Cartesian,polar, spherical polar, or cylindrical polar) and may be determined viaRTLS or the orienteering-triangulation methods with various wavelengthsor modalities, such as ultra-wideband, Bluetooth, etc. Additionally, thecoordinates may be determined using an angle of arrival or angle ofdeparture of a signal to or from the base Node, along with the distancefrom the base Node. By way of non-limiting example, this could produce adataset that correlates the coordinates of three elements with theidentities of those elements: {(0,0,0), BaseNode; (1,1,1), SmartDevice;(2,2,2), ItemOfInterest}. While this example may be used throughout thefollowing discussion, it is understood to be non-limiting, as a givenspace may include a plurality of items of interest. Note that, in someembodiments, the Smart Device itself may become a dynamic database entrywith a continuously (or periodically) updating set of coordinates. Thismay be useful in allowing a plurality of Smart Devices engaged with thesystem at the same time to interact with one another.

At step 1005, the position of the Base Node is determined relative tothe defined physical area. In exemplary embodiments, this may includeestablishing the Base Node as an origin in the coordinate system anddetermining vectors from the Base Node to boundaries and items ofinterest (i.e., the distance from the Base Node and the direction fromthe Base Node to the boundaries and items of interest). In someexamples, the Base Node may have an established reference relative to aglobal coordinate system established.

At step 1006, a Target Area is generated within a controller of theSmart Device. The Target Area may be the set of coordinates (relative tothe Base Node) within the Radio Target Area of the wireless receiver.The Target Area may be limited by physical boundaries of the givenspace, such as walls, floors, ceilings, occlusions, etc. The Target Areamay also be limited by distances that various types of signals maytravel. For example, a sensor of audio signals may not be able topractically pickup signals over a background noise level that originatemore than 1000 feet from a user position, purely as an example. In sucha case, the Target Area for such signal types may be limited to thatdimension.

At step 1007, respective positions of one or more wireless Nodes withinthe Target Area are determined. These positions may be determinedrelative to the physical Target Area or the Radio Target Area. Thedetermination may be made with reference to the dataset discussed atstep 1005, or it may be made dynamically based upon one or more BaseNodes and/or the Radio Target Area. Moreover, the determination mayadditionally be based on receipt of a wireless signal into the SmartDevice from the wireless Node. This signal may indicate a position usingthe orienteering methods described herein.

At step 1008, a user interface may be generated on the Smart Devicebased upon the pattern of digital values generated at step 1002. Theuser interface may comprise a plurality of pixels, wherein each pixelcomprises a visible color based upon the pattern of digital valuesgenerated at step 1002. For example, if the digital values were basedupon receipt of visible light into the wireless receiver (e.g., asensor), then the display may reflect a reasonably accurate colorphotograph of the Radio Target Area of the wireless receiver. If thedigital values were based upon an intensity of received light from, forexample, LIDAR, then the display may reflect a scan of the Radio TargetArea. In some embodiments, the pixel may include an intensity of energyreceived into the receiver. In this way, aspects of the Radio TargetArea characterized by an intensity of energy may be emphasized. Forexample, this may produce a LIDAR relief of an area or a heatmap of anarea.

At step 1009, an icon may be generated in the user interface. Preferablythe icon may be placed at a position relative to data quantifyingreceived energy levels. In some embodiments, the icon location in a userinterface may be indicative of a position of a Tag (Virtual orPhysical). This position may be quantified via positional coordinates,such as Cartesian Coordinates, Polar Coordinates, Spherical Coordinates,and the like. The icon may be based upon an input from a user, storeddata, quantified environmental conditions or other criteria related toan aspect of the Radio Target Area.

For example, an icon may indicate information about an Item of Interestlocated at a given set of coordinates within the Radio Target Area orDigital Radio Target Area. In another embodiment, the user may indicateon the display a position in which the user wishes to place an icon andadd information about an Item of Interest (thus creating a new entry inthe database, which may be populated with the coordinates of theindicated position). Moreover, the icon may change colors based upon thepattern of digital values. The icon may be overlaid on top of thedisplay. The icon may resemble the letter “i”, a question mark, athumbnail, or any other suitable image from a library. In someembodiments, the icon may change depending on one or more attributes ofits corresponding database entry. For example, if the icon located at(4,4,4) relates to a restaurant menu, then the icon may resemble theletter “i” or a thumbnail of a menu. On the other hand, if this databaseentry is modified so that the corresponding database entry is a message,then the icon may update to a picture of an envelope.

In some embodiments, the icon-generation step may be based upon aninquiry to a database that uses the Digital Radio Target Area as aninput. For example, upon generation of the Digital Radio Target Area, anassociated set of coordinates in one or more dimensions may begenerated. This may then be submitted to a database. An associateddisplay may be as illustrated in FIG. 9A. In some embodiments, theicon-generation step may be based upon an inquiry to a database thatuses the user's position coordinates as an input. In these embodiments,both the Digital Radio Target Area based on an RTA as well as theuniversal Radio Target Area may be included in an inquiry submitted tothe database. An associated display may be as illustrated in FIG. 9C. Insome examples, the user may have an option to limit or filter the typesof database entries that may be queried for, such as in a non-limitingsense, the existence of real-world Tags, virtual Tags, sensor datavalues and streams from a particular class of sensors and the like.

Continuing with the example from step 1004, the Digital Radio TargetArea may comprise the set of coordinates: ([1.5,10], [1.5,10],[1.5,10]). In this example, the database may return information aboutthe Item Of Interest, but not about the Base Node. The Digital RadioTarget Area may update when the Smart Device position changes, or byuser input, the Digital Radio Target Area may remain static after acertain instance in time.

Continuing with FIG. 10B, at step 1010, the icon may be positioned inthe user interface at a given position based upon coordinatesrepresentative of the position of the wireless Node or Tag in the TargetArea. This may comprise a selection of a multitude of pixels related tothe position of the wireless Node or Tag and changing those pixels fromthe digital values determined at step 1002 (check ref #) to a second setof pixels to indicate the presence of an icon. In some embodiments, theicon may be dynamically updated based upon movement of the Smart Device(and, accordingly, the wireless receiver). In some embodiments, the iconmay be permanently associated with a set of coordinates. In suchembodiments, the icon may be generated whenever a Smart Device withappropriate permissions includes in its Radio Target Area the set ofcoordinates of Nodes or Tags associated with the icon.

At step 1011, user-interactive functionality may be associated with thepixels comprising the icon. This may allow the user to “select” the iconby means of an input device (e.g., mouse, touchpad, keyboard),touchscreen, digital input, etc. Upon selection, the icon may beoperative to interact with the user in one or more ways, including:displaying a message intended for the user (by text, audio, video,hologram, etc.); requesting credentials from the user to verifypermissions (e.g., a password), displaying information about an itemassociated with the icon, prompting the user to update information aboutan item associated with the icon, etc. The user-interactivefunctionality may display static information (e.g., dimensions of theitem), display dynamic information (e.g., an alarm state or sensorinformation relating to the item; for example, if the item is arefrigerator, internal temperature may be displayed), or produce acontrol panel that allows the user to issue control commands (e.g.,remotely operating an automated apparatus by resetting an alarm state,taking remedial action based upon a sensor state as described herein,etc.) or to issue menu control commands such as to invoke a differentuser interface or screen of a user interface.

This may be useful in geospatial applications, or in procedurallygenerated activities. For example, a first user may generate apositional designation on a user interactive device, such as, forexample an augmented-reality display to leave a narrative, icon or otherinput associated with the first use. Additionally, the same or anotheruser may log positional coordinates and upload an image that could bedisplayed submitting a database query including those coordinates. Entryof the coordinates and essential credentials may provide access to thecontent associated with the positional coordinates.

At step 1012, the preceding steps may be integrated by generating adisplay comprising the user interface, the icon, and at least some ofthe associated user-interactive functionality. In embodiments, in whicha plurality of Smart Devices are themselves part of the database, thismay allow various users to send messages, images, etc. to each other.

At step 1013, detection of movement of the Smart device may cause abranch back to step 1005. Based upon that movement of the Smart Device,a defined physical area from which wireless energy is received (i.e.,the Radio Target Area based upon the Target Area) may be changed. Themovement may be detected using input from wireless communications,magnetic field sensors, an accelerometer, feature-recognition software,or other similar apparatus and algorithms. In other examples, theposition of the Smart Device may be dynamically obtained using any ofthe techniques of position determination, such as triangulation withreference nodes. Here, too, a change of position detected in this mannermay cause a branch back to step 1005. The Target Area may be based uponthe position of the Base Node, the relative positions of the wirelessNodes, and the Smart Device.

Referring now to FIG. 11, an exemplary database structure usable inconjunction with the present disclosure is shown. In this non-limitingexample, the database has five sets of information: coordinates 1101associated with an action, permissions 1102 associated with the action,the action type 1103, attributes 1104 for the action, and notes 1105.The example shown in FIG. 11 may suppose the following: theaugmented-reality system is deployed in an enclosed space, definable bya coordinate system set relative to a Base Node having an origin point(0,0,0); the enclosed space spans, in that coordinate system, ([0, 10],[0, 10], [0, 10]) (using traditional set notation; in other words, eachcoordinate can take on any number between 0 and 10, inclusive); and theRadio Target Area is ([1, 10], [1, 10], [1,10]).

The bolded entries in the database shown in FIG. 11 represent thedatabase responses to the query given by the Radio Target Area of theSmart Device, i.e., all entries having a Coordinate value within theRadio Target Area. In some embodiments, the database may sort throughall coordinates within the Radio Target Area and then return any entriesfor which the Smart Device has appropriate permissions. In otherembodiments, the database may sort through all entries for which theSmart Device has appropriate permissions and then return any entrieswith coordinates within the Radio Target Area. The latter approach maybe beneficial in circumstances in which there are numerous databaseentries with varying permissions; for example, if a database has10,000,000 entries, but a given user might only have access to five ofthose entries, sorting by permissions first may be more beneficial.

The ActionType variable may include any action for which interactivitywith an icon may be desirable. In FIG. 11, the ActionType variablesshown are Information, Message, Action, and Directions. Each of theserepresents functionalities within the scope of this disclosure. Forexample, Information may relate to information that the Smart Deviceuser may find helpful. Continuing with the shop example from FIG. 9A(check Fig Ref #), Information may include store hours, discounts,reviews, etc. Similarly, Message may be a message to the general public(e.g., an announcement), or a message tailored to a specific user. Inthe latter case, permissions may operate to ensure that only thespecific user (or set of users) may access the Message.

Action may relate to any action that a sensor, electronic device, orother apparatus connected to the database may take. For example, Actionmay include changing a temperature, measuring a temperature, turning offlights, activating an emergency sprinkler system, opening a door, etc.In some embodiments, prior to taking the Action, a password may berequested as part of the permission check.

Directions may show a user how to navigate (using, in exemplaryembodiments, orienteering methods) from the current position to adesired position. For example, upon scanning an entry on a map, virtualarrows may be generated to guide the user to a chosen store.

The ActionAttributes may have attributes based on the ActionType. Forexample, if the ActionType is Information or Message, then theActionAttributes may be a text string or a stored audiovisual filecontaining the message. Similarly, if the ActionType requires a sensoror other electronic device to take an Action, then the ActionAttributesmay include a command or subroutine to affect such an Action. In theexample shown here, the ActionType Directions comprises anActionAttribute that includes a command to the Smart Device (i.e., showdirections in the form of green arrows).

Referring to FIG. 12, an illustration of alternative methods for displayof information relating to RTA is provided. At the beginning of theprocess, a system of components which may include a smart device with auser of the smart device may be established. Amongst the variouscomponents a Home Position may be established for all the components atstep 1201. The system may proceed by establishing and initiatingtransceiving of data and information at step 1202.

In some examples, the user may be prompted to choose a desiredcoordinate system for the display at step 1203. In other examples, auser interface of the system may have a setpoint function which the usermay invoke to gain access to user settable parameters which may includethey type of coordinate system to use, such as for example Cartesian orspherical coordinates.

In still further examples, the system may decide to default to aparticular coordinate system depending on the nature of the type of dataits positional reference devices may be obtaining or providing.

At step 1204, if the coordinate system was chosen as Cartesiancoordinates, the system may utilize triangulation amongst multiplereference point transceivers. Alternatively, at step 1205 if thecoordinate system was chosen as polar coordinates, the system mayutilize positioning systems that utilize angles and distances involvedin transceiving and location. In either event, at step 1206, theposition of a Sensor attached to the smart device of the user may bedetermined. In some examples, the system may have multiple and redundantlocation system. A combination of such position determinations mayresult in superior accuracy of an aggregated position result.Accordingly, at optional step 1207, a wireless position determinationmay be performed with the smart device to establish a verification ofthe position of the Smart Device and the Sensor attached. Referring nowto step 1208, a direction that the sensor is facing in may bedetermined. Although there may be a number of different manners ofdetermining orientation as have been described herein, in an example,the orientation may be determined based upon wireless transmissionand/or wireless verification.

Referring now to step 1209, an energy-receiving Sensor included in theSmart Device or in logical communication with the Smart Device may beused to quantify energy levels perceivable at the position and in thedirection of the Smart Device. The resulting quantification may dependon aspects of the Sensor device, but the resulting data may quantify acharacteristic for the RTA.

In some embodiments, an optional step 1210 may be performed by anelement of the system such as the smart device or a server incommunication with the Smart Device. The element of the system maycompare one or more of position information, orientation information andthe image data itself to calculate an estimate of whether the RTA anglehas changed for the sensing element.

In general, at step 1211, the RTA of the Sensor device used to capturethe image in step 1209 may be quantified. In an optional step 1212,coordinates relating to the instant RTA of the image may be established.In some examples, this may relate to a range of three-dimensionalcoordinates that are addressed by the RTA of the Sensor element. Ingeneral, at step 1213, the system may look up, or in some casesgenerate, location coordinates for Tags that are determined to be withinthe quantified RTA. In some database systems that the system may haveaccess to, real-world or virtual-world tags may be tracked in acoordinate system with a certain origin.

If the current origin established at step 1201 is offset from aparticular database related origin, then one or both the coordinatesystem values may be converted to each other to align their respectiveorigins. At step 1214, the Tags in an aligned coordinate system may havetheir positions compared to the current RTA and a selection for the setof Tags that are within the RTA may be made.

In some alternative examples, a display of all Tags that are authorizedfor access to the user regardless of whether they are in the RTA may bemade using associated aligned coordinates as discussed in reference tostep 1213.

Referring now to step 1215, in an example, the Smart Device of the usermay be used to generate and display a user interface to the user basedupon the captured image and the associated tag icons within the RTA.These associated Tag icons may have at least the functionality as hasbeen discussed in reference to FIGS. 10A and 10B.

Referring now to FIG. 13, a Smart Device 1301 is illustrated within awireless communication area (WCA) 1302. The extent of the particular WCA1302 may be defined according to a select bandwidth and/or a particularmodality of the wireless communication the Smart Device 1301 uses totransmit and receive information.

For example, bandwidths may include those associated with UWB, Wi-Fi,Bluetooth, ANT, ultrasonic, infrared, and cellular modalities ofcommunication. In general, unless otherwise constrained by physicalmodification such as the use of a directional antenna, or the presenceof radio frequency interference from a physical object (such as objectswith significant metallic content; objects with high water content;electrical fields; etc.), a WCA 1302 may include spherical area(s)emanating from one or more transceivers and/or transceiver antennasoperated by the Smart Device 1301.

As discussed extensively herein, and in patent applications referencedby this application, the location of the Smart Device 1301 may bedetermined based upon wireless communication to and/or from the SmartDevice 1301; and described via a coordinate system, such as viageneration of Cartesian coordinates, or other coordinates such as: polarcoordinates, spherical coordinates, and cylindrical coordinates.Modalities of wireless communications that may be referenced to generatelocation coordinates may include one or more of: RTT (round trip time),time of flight, RSSI (received signal strength indicator); angle ofarrival, angle of departure, and other methods, equipment and modalitiesas have been described herein.

With the location of the Smart Device 1301 determined, a location of theWCA 1302 may be extrapolated based upon the location of the Smart Deviceand a range or transceiving distance the Smart Device may be capable of.

According to the present invention, a portion of the WCA 1302 may beselected as a radio target area (RTA) 1312 from which the Smart Device1301 may receive specific bandwidths of electromagnetic radiation. Inpreferred embodiments, the RTA 1312 may include a frustum expandingoutward in a conical shape from one or more energy-receiving Sensors1309 included in the Smart Device 1301. The frustum shaped RTA 1312 mayoverlap with a portion of the generally spherically shaped WCA 1302.Other shapes for an RTA 1312 are also within the scope of thisspecification.

In some embodiments, a shape of the RTA 1312 may be based upon receivingcapabilities of the one or more energy-receiving Sensors 1309incorporated into or in logical communication with the Smart Device1301. For example, an energy-receiving Sensors 1309 with a chargecoupled device (CCD) or complementary metal oxide semiconductor (CMOS)receiver may have a single plane receiving surface and be best matchedwith a frustum of a generally pyramidal or conical shape. Whereas anenergy-receiving Sensors 1309 with multiple receiving surfaces (e.g.,with multiple CCD and/or CMOS devices) may be arranged to enable a morecomplex shaped RTA 1312.

In some preferred embodiments, a direction of interest 1311 mayintersect the RTA 1312. As discussed herein, the direction of interest1311 may be represented by a ray or vector 1311A and 1311B. In addition,the direction of interest 1311 may be represented as a direction ofinterest area, such as a frustum defined by multiple rays or vectors. Invarious embodiments, the direction of interest 1311 area may encompassthe RTA 1312 or be a subset of the RTA 1312.

A direction of interest 1311 may be determined for example via themethods and devices described herein and in referenced patentapplications and may be associated with a direction based upon a ray orvector indicative of a direction of interest 1311, a direction basedupon a magnetic field sensor, an accelerometer, a light beam,correlation between two Tags or Nodes, Agent gestures, or other SmartDevice recognized apparatus and/or method.

One or more transceivers 1303-1305 (typically included within a SmartDevice, Tag, or Node) may be located within an area defined by the RTA1312. According to the present disclosure, a position of the transceiver1303-1305 may be determined and a user interactive mechanism may begenerated at a position of the transceiver 1303-1305 within a graphicaluser interface emulating aspects of the RTA 1312 on the Smart Device1301 or another user interactive interface screen (not shown, andperhaps at a site remote to the RTA 1312).

According to the present disclosure, some portion of the RTA 1312 (whichmay include the entirety of the RTA 1312) may be portrayed on an Agentinterface 1310, including, in some embodiments, a human-readablegraphical user interface (GUI). The interface 1310 may include arepresentation 1313 of a particular level of electromagnetic energyreceived via the energy-receiving Sensors 1309 and associated with aparticular area of the RTA 1312. For example, energy levels of aninfrared wavelength that has emanated from or reflected off of an itemin the RTA 1312 and received via an infrared receiver in the SmartDevice 1301 may be used to generate a heat map type interface display.Similarly, energy that has emanated from or reflected off of an item inthe RTA 1312 in the 400 nm to 700 nm range and been received via acharge-coupled/or CMOS image sensing device in the Smart Device 1301 maybe portrayed as a human visible image of items in the area included inthe RTA 1312.

Other embodiments may include a point cloud derived from electromagneticenergy bouncing off of or emanating from items included in the RTA 1312or a series of polygons generated based upon a LIDAR receiver in theSmart Device 1301. An Agent interface 1310 may be presented in amodality understandable to an Agent type. For example, an interfacepresented to a UAV or UGV may include a digital pattern and an interfacepresented to a human Agent may include multiple pixels or voxelsgenerating a pattern visible to a human being.

The wireless location methods and apparatus described herein may bedeployed in conjunction with one or more Transceivers 1303-1305 or Tagsand/or Nodes 1306-1308 located with the WCA 1302 to generate locationcoordinates for the one or more Transceivers 1303-1305 or Tags and/orNodes 1306-1308. A controller or other device operating a processor maydetermine which one or more Transceivers 1303-1305 or Tags and/or Nodes1306-1308 located within the three-dimensional space included in the RTA1312 based upon a) the location of the one or more Transceivers1303-1305 or Tags and/or Nodes 1306-1308; and b) the location of areaincluded in the RTA 1312.

In another aspect of the present disclosure, in some embodiments, someenergy levels may not be represented in the Agent interface 1310. Forexample, in some embodiments, energy levels reflected off of aparticular item may not be included in the Agent interface 1310. Otherembodiments may only represent energy levels that have reflected off ofselected items within the RTA 1312 thereby emphasizing the presence ofthe selected items and ignoring the presence of other items within theRTA 1312.

As described above, some portion of the RTA 1312 may be portrayed on anAgent interface 1310, including, in some embodiments, a human readablegraphical user interface (GUI), as a point cloud derived fromelectromagnetic energy bouncing off of or emanating from items includedin the RTA 1312 or a series of polygons generated based upon a LIDARreceiver in the Smart Device 1301. An example of such a representationis shown in FIG. 14. In this example, the GUI includes a human visualimage 1401 of an RTA 1400 overlaid with a series of polygons 1402generated based upon a LIDAR receiver in the Smart Device. The LIDARsensor illuminates the RTA 1400 with laser light and then measures thereflection with a sensor. The resulting polygons 1402 representdifferences in laser return times, which provides a topographicalrepresentation of objects in the RTA 1400.

In this example, Virtual Tags 1403 and 1404 are created by the SmartDevice by methods described herein and icons may be present on the GUIto identify the position of the Virtual Tags 1403 and 1404. The VirtualTags 1403 and 1404 may, for example, represent various locations ofinterest in the RTA 1400, such as an object of interest or an exit orentrance The icons associated with the Virtual Tags 1403 and 1404 may beengaged or “clicked” or otherwise activated to be made operational; forthe Smart Device to receive (e.g., retrieved from a database) additionalinformation associated with the object or location of interest.

For example, if the object of interest is a statue, clicking on the iconassociated with the Virtual Tag 1403 associated therewith may provideinformation regarding the statue, such as the history, origin, and thelike. If, for example, the Virtual Tag 1404 is associated with an exitof the room, clicking the Virtual Tag may provide information on what ispresent in the adjacent room, or where the Smart Device is in relationto exiting the building, or any other desired information.

In some embodiments, mathematical data associated with a LIDARrendering, such as parameters of triangles formed by various LIDARpoints 1405-1406 within an associated RTA may be stored and a relativeposition of a smart device with the RTA 1400 may be determined basedupon the recognition of similarities of the LIDAR point 1405-1406patterns. A resolution of laser scanning involved in generation of databased upon LIDAR techniques may influence a number of date points withina selected RTA, but in general, pattern recognition and determination ofan orientation of a smart device based upon LIDAR data may be much moreefficient, than, for example image data based pattern recognition. Inaddition, the LIDAR based patterns may be formed in a “fingerprint” ofan RTA, wherein it would be very rare, if not impossible to replicatethe LIDAR point patterns at two disparate locations. Therefore,recognition of a point pattern may be used to identity a location of aparticular RTA.

FIGS. 15-15A—Location Determination with Wireless Means Referring now toFIG. 15, a basic illustration is provided for description of theprinciples involved in determining the location of a radio receiver inthree dimensional space based on its interaction with radio transceiverswhose location are precisely known. In principle, to determine alocation in three dimensional space at least three radio transceiverslocated at different locations is required. As may be described in moredetail following, it is possible to obtain improvements in operations byuse of additional transceivers. For illustration purposes, fourtransceivers are included in the illustration of FIG. 15. A firsttransceiver 1501 is illustrated as a “base position” with exemplarycartesian position coordinates of 0, 0, 3000. In an example, the homeposition of 0, 0 may be a relative position along a floor of a structureand as the transceiver may be located upon a ceiling of a structure, theheight of 3000 mm may represent a 3 meter elevation of the firsttransceiver 1501. Other transceivers may also be located on a ceilingsuch as the second transceiver 1502 at position 5000, 0, 3000. The thirdtransceiver 1503 may be located at position 5000,15000,3000 in anexample. And a fourth transceiver 1504 may be located at position 0,15000, 3000. A radio position tag may be located at a first positionlocation 1505.

There may be numerous techniques that the system of transceivers and aradio position tag may be used with to determine positions bytriangulation. In some examples, a distance from a transceiver to theradio tag may be determined by the measurement of the strength of thesignal which is related to the distance the signal has travelled. Inother examples, timing signals may be used to determine the precise timeit takes for a signal to transit the distance. In still furtherexamples, the angles of arrival of signals from or to the transpondermay be used in combination to determine a location. Thus, the firstposition location 1505 may be determined by making at least threedeterminations of distance or angles between a radio position tag atfirst position location 1505 and the four transceivers (1501-1504).

It may be appreciated, that all four of the determinations may be madeand can be used to calculate a position. In a real world environment,there may be a number of factors that lead to difficulties in obtaininga position or angle measurement with precision. In a first example, thepositioning system may be installed in an interior location with walls,equipment, occupants, and other elements formed of materials that mayinteract with the radio signals (or other position signals such asinfrared, laser, and ultrasonic signals as non-limiting examples.Scattered signals from walls may cause multiple signal paths which maycause confused signal determination. Depending on the location of thefirst position within the space of interest, one or more of the signalpaths from the transceivers to the first position tag may experiencesignal degradation from one of the causes.

The positioning system may utilize all four of the signals to determinethat one of the three signals could be omitted to improve the “goodness”of a position determination. Both the instantaneous signal pathdetermination as well as the time progression of a signal pathdetermination may factor into the rejection of a particular signal pathmeasurement. Accuracy of position determination may be improved bysampling numerous times. In some examples, ultrawide band transmissionprotocols may be used to make position determination with relatively lowenergy consumption and good signal to noise aspects. Ultrawidebandprotocols may use extremely narrow (nanosecond) pulses which may allowfor improved discrimination of multipath signal arrivals. Many differentfrequencies may be utilized for high degrees of sampling which canimprove the accuracy.

Referring again to FIG. 15, a second position location may include anagent with two tags located at locations 1506 and 1507. The accuracy ofdetermination of a second position may improve by the use of multipletags at the different locations 1506 and 1507 whose aggregate positiondetermination may be used to determine a second position location. Asmay be observed in the dashed lines and the complex dot dashed linethere are individual signal paths from each of the four transceivers tothe two tags. Here too the increased number of position signal pathsallow for statistical improvement of an average location and also allowfor the ability to statistically process signals to eliminate poormeasurements or to average out poor measurements by acquiringcollections of signals over time.

In some examples, advantages may be obtained by establishing systemswith larger numbers of transceivers. An example may be found inreference to FIG. 15A where an assembly of eight position transceiversare configured. The eight discrete locations are illustrated again at aheight of 3000 mm such as a system mounted on a ceiling where a floormight be located at 0. A base coordinate location may be defined as thebase position 1501A. The other position transceivers (1502A-1508A) maybe located at other discrete locations also indicated at a height of3000 mm for example. Now, a position tag 1510A may transmit or receivesignals to or from the 8 transponders and each of these paths may beused to determine the location. The statistical combination ofadditional transponder paths may implicitly improve the accuracy ofmeasurements. This may be especially true if the system is trained withknown positions from time to time, which may particularly help in theremoval of static or systematic errors such as may occur in the locationof a given position transponder. In real use conditions, there may bedynamic aspects of the environment which may affect performance. Forexample, a human occupant of a space or an object that is placedtransiently into a location may create an obstruction 1520 A to line ofsight path measurements such as the illustrated obstruction 1520A. Thelocation determination system may detect the presence of an obstructionby a number of means such as a reduced signal strength over historicallevels relative to other signal strengths at other transceivers. In someexamples, interference by an obstruction 1520A may result in a distanceor angle determination of the signal path D5 that is entirelyinconsistent with the determination of position from some or all of theother paths D1-D4 and D6-D8.

A combination of three or more transceivers may be networked to form aself-verifying array of nodes. Examples of self-verifying arrays ofnodes are discussed in detail in another application of the applicant:U.S. application Ser. No. 16/775,223 filed Jan. 28, 2020 and entitled“SPATIAL SELF-VERIFYING ARRAY OF NODES” the contents of which is herebyincorporated by reference it its entirety. In some examples, a number oftransceivers such as the eight transceivers illustrated in FIG. 15A mayform an array of nodes which communicates between itself and forms itsown control aspects. The array of nodes may dynamically assess thenature of signals exchanged.

In some examples, the array of nodes may include processing andalgorithms that optimize communication aspects for the array of nodesrelative to individual tags within the communication range of the arrayof nodes. As illustrated, one of the paths may be impeded by aspectssuch as a physical blockage that may eliminate the path as a line ofsight path. The array of nodes may sense the blockage for a particulartag and disable that path while the blockage is present. In dynamicenvironments, the blockage may come for example from a user walkingaround in the environment. In other examples, the tag may move in such amanner that a path becomes blocked. In these type of cases, the node mayadjust to optimize those paths that are good. For example, when a seriesof Ultrawideband pulses is used in determining location aspects, theduty cycle of various nodes may be adjusted so that more samples areacquired with node to tag paths that are good.

In some examples, the array of nodes may be dynamic in the sense thatnew nodes may enter into the array of nodes when they become present inthe geographic region of the array of nodes. In some examples, the arrayof nodes may recognize a new node and determine that it knows itsposition to an acceptable degree of accuracy to be utilized insupporting location determination of tags that communicate with thearray of nodes. In some examples, the self-verifying array of nodes maybe established at fixed locations on a structure or infrastructure alongwith movable node components. The movable node components may allow thearray of nodes to dynamically address requests for service from tagswithin the transmission by extending a movable node to a region where itmay better communicate with the tag. An array of nodes may function toimprove location and orientation determinations of tags.

Referring to FIG. 15B, a representation of a self-verifying array ofnodes space 1500B. In this embodiment, space 1500B may includeStructures 1511B and 1512B. Structures 1511B and 1512B may have avariety of different characteristics that may impact the performance ofself-verifying array of nodes 1510B. For example, Structures 1511B and1512B may be physically closed (e.g., walls, solid Structures) orpartially closed (e.g., shelves). Structures 1511B and 1512B may alsocomprise solid materials (which may be stored for example at aconstruction site), equipment such as automobiles, tractors, backhoes,and the like. Accordingly, the presence of these Structures may changethe transmission characteristics of a wireless network (e.g.,Bluetooth). Some Structures may block signals, impede signals, orpartially impede signals. For example, shelves may have physical regionsthat block and other regions that are fully transmissive.

Shelves may provide an example in which the Structures in the space1500B may have dynamic characteristics. Such dynamic characteristics maymake self-verifying arrays (and corresponding spatial schema) moreuseful than traditional mapping methods. For example, if a load of metalpiping is brought in and placed upon the shelves, a region that wascompletely transmissive may become impeded to a degree. Thesecharacteristics may create different operational characteristics forself-verifying arrays.

In another sense, a shelf may hold a combination of both fixed andmobile devices that comprise a self-verifying array in the space at somegiven time. This may provide more accurate and more dynamic coverage forthe schema. For example, the space 1500B may be interspersed with anassembly of fixed (or roughly fixed) network Nodes that form a gridpattern (as an example) to ensure that a minimal self-verifying arraymay be established that covers the entire space 1500B. This minimalnetwork may be supplemented with “migrant” Nodes that are moved into thespace 1500B and become part of the self-verifying array of Nodes 1510B.From a signal coverage perspective, more participants may improvecharacteristics. However, more participants may increase informationtraffic levels, and a control formalism that limits bandwidthdifferentially to different network participants may be necessary insome examples.

In FIG. 15B, an example of a space 1500B with shelving units that makeup Structures 1511B and 1512B is illustrated. The space may have a“global” reference point 1504B for positioning. There may be fixedwireless communication Nodes 1501B, 1502B, and 1503B (for this example,all Nodes are at least compliant with Bluetooth 5.1 and transmit atleast as Bluetooth radio transmitters; however, this deployment ismerely illustrative). The fixed wireless communication Nodes 1501B-1503Bmay also include other aspects/components to them such as an integratedcamera system. The integrated camera system may provide a visualperspective of a portion of the space that its corresponding wirelessradios may cover. In a self-verifying array, Nodes may be collocated orlocated relative to a Sensor, such as an image-capture device. Based ona known set position of the Sensor relative to the Node, the Node maytransmit information captured by the Sensor to other Nodes. Accordingly,a Node out of both Sensor and radio range of another Node may stillreceive data from the Sensor through the array. The data from the Sensorreflects a range of data in which a physical characteristic isquantified or capable of being quantified by the Sensor. For example, aSensor may be an image-capture device, limited in range by bothwavelength of image capture (e.g., limited to infrared) and spatialrange (e.g., field of view of the image-capture device). This may beparticularly desirable in embodiments in which the self-verifying arrayis deployed in or adjacent to an environment having a characteristicadverse to a Sensor. For example, the low temperatures found in acommercial freezer may impair operation of certain Sensors.Temperature-resistant Sensors may be collocated with Nodes within thefreezer, while temperature-vulnerable Sensors (including Sensors capableof detecting conditions within the freezer) may be collocated outsidethe freezer. Through the self-verifying array comprised of these Nodes,data from the Sensors may be freely transferred among the Nodes,including through fiber optic communication throughout the freezer. Itmay be desirable to deploy spectrometers and hydrometers in thisfashion. Moreover, redundant Nodes may be able to redirect Sensorreadings from one Node to a base Node, especially in scenarios when anoptimal Node pathway may be obstructed, such as by shelving.

The space 1500B may also include other fixed Nodes, such as Node 1523B,that may not have cameras included. Node 1523B may be important toensure that regardless of a makeup of migrant communication Nodes, fixedwireless communication Nodes may be able to form a completeself-verifying array of Nodes space 1500B in the absence of items thatblock radio transmissions. There may also be migrant communication Nodes1520B-1522B affixed on packages, materials, or other items that may beplaced and/or stored upon the shelving units.

In some examples, at least a subset of the self-verifying array of Nodesparticipant Nodes may communicate periodically. The various aspects ofdata layer communications as have been discussed may occur between theNodes of the network. At a base level, at least a subset of theBluetooth transmitters may periodically transmit information such astheir unique identifiers, time stamps, known positions and the like. Insome embodiments, Nodes may transmit between each other or to a baseinformation about variables between the Nodes, such as computeddistances or angles between the Nodes. A Node may receive transmissionsfrom other transmitters and may store the transmissions. In someexamples, a Node may act as a repeater by receiving a transmission andthen retransmitting the received transmission. A Node acting as arepeater may then take various actions related to the data involved. Inan example, the Node may effectively just stream the data where nostorage of any kind is made. Alternatively, a Node may store thetransmission, then retransmit the transmission (immediately or after adelay) and then delete the stored data. In other examples, a repeaterNode may store a received transmission and then retransmit thetransmission either for a stated number of times, or until some kind ofsignal is received after a transmission. Thereafter the Node may alsodelete the data. In some examples, a Node may store data from anincoming transmission and take the various retransmission actions ashave been defined, but then not delete data until its data store isfilled. At that point, it may either delete some or all of the storeddata, or it may just overwrite stored data with new incoming data andthen clean up any remaining data with a deletion or other process.

When a Node acts as a repeater it may receive data and then merelyretransmit the data. Alternatively, a repeater Node may either use thetransmission of data or the time during the transmission to acquire andcalculate its position and potentially the position of othertransmitters in range. During retransmission of the received data, itmay also include in the transmission calculations of its own positionrelative to other transmitters, calculations of other transmitterpositions relative to itself, calculations of its own and othertransmitter positions relative to an origin, and the like. It may alsoinclude other information such as a time stamp for the calculation ofpositions.

The combined elements of a self-verifying array of Nodes may be operatedin a way to optimize power management. Some of the network Nodes andtransmitting elements may operate in connection with power-providingutility connections in the Structure. Other network Nodes may operate onbattery power. Each of the Nodes may self-identify its power source, andeither at a decision of a centralized controller or by a cooperativedecision making process, optimized decisions may be taken relative todata transmission, low power operational modes, data storage and thelike. In some examples, where multiple Nodes provide redundant coverageand provide information to a central bridge acting as a repeater, theNodes may alternate in operation to share the power-draw on individualNodes. For example, if one of these Nodes is connected to a utilitypower source, that Node may take the full load. The battery-poweredelements may have charge-level detectors and may be able to communicatetheir power-storage level through the network. Accordingly, anoptimization may reduce traffic on the lower battery capacity Nodes.

In some examples of operations, a transmitting Node may transmit amessage for a number of redundant cycles to ensure that receivers have achance to detect the message and receive it. In low power operatingenvironments, receivers may transmit acknowledgements that messages havebeen received. If a base unit of the network acknowledges receipt of themessage, control may be transferred to the base unit to ensure that themessage is received by all appropriate network members. Messagereceivers may make a position determination and broadcast their positionif it has changed. A self-verifying array of Bluetooth receivers mayprovide one of a number of Transceiver network layers where othercommunication protocols based on different standards or frequencies ormodalities of transmission may be employed, such as WiFi, UWB, Cellularbandwidth, ultrasonic, infrared and the like. A Node that is a member ofdifferent network layers may communicate and receive data between thedifferent network layers in addition to communicating through aBluetooth low-energy self-verifying array.

Controller and Node Aspects

Referring now to FIG. 16A, an automated controller is illustrated thatmay be used to implement various aspects of the present invention invarious embodiments, and for various aspects of the present invention.Controller 1600 may be included in one or more of: a wireless tablet orhandheld smart device, a server, an integrated circuit incorporated intoa Node, appliance, equipment item, machinery, or other automation. Thecontroller 1600 includes a processor unit 1602, such as one or moresemiconductor based processors, coupled to a communication device 1601configured to communicate via a communication network (not shown in FIG.16A). The communication device 1601 may be used to communicate, forexample, with one or more online devices, such as a smart device, aNode, personal computer, laptop, or a handheld device.

The processor unit 1602 is also in communication with a storage device1603. The storage device 1603 may comprise any appropriate informationstorage device, including combinations of digital storage devices (e.g.,an SSD), optical storage devices, and/or semiconductor memory devicessuch as Random Access Memory (RAM) devices and Read Only Memory (ROM)devices.

The storage device 1603 can store a software program 1604 withexecutable logic for controlling the processor unit 1602. The processorunit 1602 performs instructions of the software program 1604, andthereby operates in accordance with the present invention. The processorunit 1602 may also cause the communication device 1601 to transmitinformation, including, in some instances, timing transmissions, digitaldata and control commands to operate apparatus to implement theprocesses described above. The storage device 1603 can additionallystore related data in a database 1605 and database 1606, as needed.

Referring now to FIG. 16B, an illustration of an exemplary wireless Nodeconfigured with a transceiver 1624 to wirelessly communicate via one ormore wireless communication Modalities, including a bandwidth andprotocol, such as the Bluetooth 5.1; BLE5.1; Wi-Fi RT and/or GPSstandard is illustrated. As discussed, many different Modalities ofwireless technology may be utilized with the content presented herein,but a BLE5.1 “radio” module is an interesting example since itsstandards provide for angle of arrival (AoA) capability as well as angleof departure (AoD) and a distance determination based upon a timingsignal. With AoA/AoD a designed antenna array 1625 can be used by atransceiver 1624 to measure a phase shift amongst multiple antennaelements to estimate distance differences between the antennas and toextract an angle from the antenna array to the source of radiation. ABLE5.1-consistent multichip transceiver 1624 may include circuitry andsoftware code to perform the acquisition of data and determine the angleof arrival in some examples. In other examples, a BLE5.1-consistentmultichip transceiver 1624 may control the acquisition of data from anantenna array while streaming the data to off module processingcapabilities. The BLE5.1-consistent Node 1610 may contain functionalblocks of circuitry for peripheral 1620 control. The peripherals mayinclude a connection to external host controllers/MCUs 1621. Theperipheral 1620 control may also interact with peripheral and IoTSensors and other devices 1622.

The BLE5.1-consistent Node 1610 may include a processing element 1623which may have its own memory of different types as well as capabilitiesfor encryption of data. The BLE5.1 consistent Node in the transceivermodule 1610 may also have Transceiver 1624. This circuitry may includeBaseband and RF functions as well as control the AoA functions and theself-verifying array functions. The Bluetooth transceiver 1624 mayreceive signals through an on-module antenna 1625 or an external antennaor array of antennas may provide external RF input 1626. TheBLE5.1-consistent Node 1610 may include functional circuitry blocks forcontrol of Security functions 1627, crypto-generation, random numbergeneration and the like. The BLE5.1-consistent Node 1610 may includefunctional blocks for power management 1628.

The BLE5.1-consistent Node 1610 may be operative for quantification oftemperature aspects of the Node of the transceiver module 1610,battery-control functions and power-conversion functions. An externalpower source 1633 may be included to provide electrical energy to apower management unit 1628 which, in some examples. may be from abattery unit, or a grid connected power supply source in other examples.The BLE5.1-consistent Node 1610 may include functions for control oftiming and triggering 1629. In a related sense, the BLE5.1-consistentNode 1610 may include functions for clock management 1630 within themodule. The BLE5.1-consistent Node 1610 may also include circuitelements that are always-on 1631 to allow external connections 1632 tointeract with the device and perhaps awake it from a dormant state.There may also be other customized and/or generic functions that areincluded in a BLE5.1-consistent Node 1610 and/or multichip module.(missing #1621 in spec)

Referring now to FIG. 16C, a Node 1650 included in a higher orderdeployment assembly is illustrated. A deployment Node 1650 may be inlogical communication with one or more of: sensors, customized controlcommands, antenna array designs and the like.

A Node 1650 may include multiple antennas or antenna arrays. Asdescribed previously, the Node 1650 may include a transceiver module1610, and in some examples, the transceiver module may includeBluetooth-adherent aspects. Communications received via an antenna1651-1656 may be directly ported into the transceiver module 1610.Embodiments may also include routing particular antenna/antenna arrayoutputs to the transceiver module 1610 in a controlled and timedsequence. A processing Module 1670 may coordinate a connection of theNode 1650 to external peripherals.

In some examples, circuitry 1680 to logically communicate with one ormore of: a Peripheral, a data Connection, Cameras and Sensorscontrollers, and components to perform data and image acquisition ofvarious kinds, or it may interface external components with the Node1650.

The Node 1650 may also include its own power management unit 1660 whichmay take connected power or battery power or both and use it to provethe various power needs of the components of the assembly. The Node 1650may have its own processing modules 1670 or collections of differenttypes of processing functions which may have dedicated memory components1671. In some examples, specialized processing chips of various kindssuch as Graphical Processing Units and fast mathematics functioncalculators as well as dedicated artificial intelligence processingchips may be included to allow the Node 1650 to perform variouscomputational functions including location determination of wirelesslyconnected devices amongst other functions. There may be numerous otherfunctions to include in a Node 1650 and alternative types of devices toperform the functions presented herein.

In some examples as illustrated in FIG. 16D antenna arrays 1690, 1691may be assembled into a “Puck” shown as Node 1650 wherein the antennaarrays are configured with antenna designs which have directionalaspects to them. Directional aspects may mean that the antennas may besensitive to incident radiation coming from a certain direction but notsensitive to radiation coming from a different direction. Antenna arrays1690, 1691 may include antennas that may have maximized signals for aparticular incident waveform, the identification of which antenna mayprovide or supplement angle of incidence calculations.

A directional antenna may include, for example, an antenna with RFshielding over some portion of an antenna's circumference. For example,2700 (or some other subset of a 3600 circumference of an antenna), or anantenna array may have RF shielding to block and/or reflect back an RFsignal towards the antenna-receiving portion. Other directional antennasmay include a shield blocking less than 3600 of RF transmissions thatrotates around a receiving portion of an antenna and only receives RFcommunications from a direction of an opening in the shield. Shieldedantennas may provide improved determination of a direction from which awireless transmission is being received from, since RF noise is blockedfrom a significant portion of a reception sphere.

Referring now to FIG. 17, a block diagram of an exemplary Smart Device1702 is shown. Smart Device 1702 comprises an optical capture device1708 to capture an image and convert it to machine-compatible data, andan optical path 1706, typically a lens, an aperture, or an image conduitto convey the image from the rendered document to the optical capturedevice 1708. The optical capture device 1708 may incorporate a CCD, aComplementary Metal Oxide Semiconductor (CMOS) imaging device, or anoptical Sensor 1724 of another type.

A microphone 1710 and associated circuitry may convert the sound of theenvironment, including spoken words, into machine-compatible signals.Input facilities may exist in the form of buttons, scroll wheels, orother tactile Sensors such as touch-pads. In some embodiments, inputfacilities may include a touchscreen display.

Visual feedback to the user is possible through a visual display,touchscreen display, or indicator lights. Audible feedback 1734 may comefrom a loudspeaker or other audio transducer. Tactile feedback may comefrom a vibrate module 1736.

A magnetic force sensor 1737 such as a Hall Effect Sensor, solid statedevice, MEMS device or other silicon based or micro-electronicapparatus.

A motion Sensor 1738 and associated circuitry converts motion of thesmart device 1702 into a digital value or other machine-compatiblesignals. The motion Sensor 1738 may comprise an accelerometer that maybe used to sense measurable physical acceleration, orientation,vibration, and other movements. In some embodiments, motion Sensor 1738may include a gyroscope or other device to sense different motions.

A location Sensor 1740 and associated circuitry may be used to determinethe location of the device. The location Sensor 1740 may detect GlobalPosition System (GPS) radio signals from satellites or may also useassisted GPS where the mobile device may use a cellular network todecrease the time necessary to determine location. In some embodiments,the location Sensor 1740 may use radio waves to determine the distancefrom known radio sources such as cellular towers to determine thelocation of the smart device 1702. In some embodiments these radiosignals may be used in addition to GPS.

Smart Device 1702 comprises logic 1726 to interact with the variousother components, possibly processing the received signals intodifferent formats and/or interpretations. Logic 1726 may be operable toread and write data and program instructions stored in associatedstorage or memory 1730 such as RAM, ROM, flash, SSD, or other suitablememory. It may read a time signal from the clock unit 1728. In someembodiments, Smart Device 1702 may have an on-board power supply 1732.In other embodiments, Smart Device 1702 may be powered from a tetheredconnection to another device or power source.

Smart Device 1702 also includes a network interface 1716 to communicatedata to a network and/or an associated computing device. Networkinterface 1716 may provide two-way data communication. For example,network interface 1716 may operate according to the internet protocol.As another example, network interface 1716 may be a local area network(LAN) card allowing a data communication connection to a compatible LAN.As another example, network interface 1716 may be a cellular antenna andassociated circuitry which may allow the mobile device to communicateover standard wireless data communication networks. In someimplementations, network interface 1716 may include a Universal SerialBus (USB) to supply power or transmit data. In some embodiments, otherwireless links may also be implemented.

As an example of one use of Smart Device 1702, a reader may scan somecoded information from a location marker in a facility with Smart Device1702. The coded information may include for example, a hash code, barcode, RFID, or other data storage device. In some embodiments, the scanmay include a bit-mapped image via the optical capture device 1708.Logic 1726 causes the bit-mapped image to be stored in memory 1730 withan associated time-stamp read from the clock unit 1728. Logic 1726 mayalso perform optical character recognition (OCR) or other post-scanprocessing on the bit-mapped image to convert it to text. Logic 1726 mayoptionally extract a signature from the image, for example by performinga convolution-like process to locate repeating occurrences ofcharacters, symbols, or objects, and determine the distance or number ofother characters, symbols, or objects between these repeated elements.The reader may then upload the bit-mapped image (or text or othersignature if post-scan processing has been performed by logic 1726) toan associated computer via network interface 1716.

As an example of another use of Smart Device 1702, a reader may recitewords to create an audio file by using microphone 1710 as an acousticcapture port. Logic 1726 causes audio file to be stored in memory 1730.Logic 1726 may also perform voice recognition or other post-scanprocessing on the audio file to convert it to text. As above, the readermay then upload the audio file (or text produced by post-scan processingperformed by logic 1726) to an associated computer via network interface1716.

A directional sensor 1741 may also be incorporated into Smart Device1702. The directional device may be a compass and be based upon amagnetic reading or based upon network settings. The magnetic sensor mayinclude three axes of magnetic sensitive elements and may also becoupled with an accelerometer in the directional sensor 1741.

A LiDAR sensing system 1751 may also be incorporated into the SmartDevice 1702. The LiDAR system may include a scannable laser light (orother collimated) light source which may operate at nonvisiblewavelengths such as in the infrared. An associated sensor device,sensitive to the light of emission may be included in the system torecord time and strength of returned signal that is reflected off ofsurfaces in the environment of Smart Device 1702. Aspects relating tocapturing data with LiDAR and comparing it to a library of stored data(which may be obtained at multiple angles to improve accuracy) arediscussed above.

Referring now to FIG. 18A an exemplary user interactive interface 1800Ais illustrated with features that are conducive to enabling the methods,processes and apparatus deployment described herein. The userinteractive interface 1800A includes various user interactive areas1801-1809. Each interactive area 1801-1809 may be activated to have anassociated controller become operative to perform a function. Theinteractive areas 1801-1809 may be integrated into image data. Imagedata may be from a file, such as a two dimensional image of a site plan,floor design, or site layout, and/or energy levels received by a sensorin a smart device. For example, activation of an icon 1801 on wall maybe integrated with image data of the wall 1807 and allow a user tocontrol functionality included in the portion of a facility proximate tothe wall, such as, by way of example, powering on/off an appliance ordevice; connecting to application that controls facilitiesinfrastructure; adjusting volume; adjust a display option; or otherfunctionality. Similar user interactive areas are linked to control ofother items of equipment and functionality appropriate for an associateditem of equipment.

IoT sensors may be associated with transceivers and combined into a unitfor quantifying conditions present at the location, such as, for examplea Shade™ Multi Sensor unit 1802 and/or a Shade Action Box 1803. A ShadeAction Box 1803 will be associated with interactive controls linked tothe user interface that allow for wireless of a structure aspect, suchas a water turn off valve, a door lock, HVAC control, electrical on/offand the like within the facility or infrastructure.

Specific icons each associated with a disparate IoT sensor may bedisplayed with associated logos in a specified user interactive area1809. Icons 1809 a, 1809 b may be associated with one or more of: fire,power, power surge, power outage, temperature, water, humidity,vibration and almost any condition quantifiable via an electronic orelectromechanical sensor.

Interactive areas 1804-1806 may provide user control of aspects of theuser interactive interface. For example, a calibration interactive area1804 provides a user with control of functionality that allows the userto align A/R sensor data, such a visible light wave data and/or infraredimage data to be aligned with an icon 1801-1803 or other designated userinteractive area.

Another control area 1808 allows a user to which assets are displayed inthe user interactive interface 1800A. Options illustrated include allassets, fixed assets, and mobile assets. Fixed assets may be associatedwith an IoT senor that is not combined with a transceiver and/or avirtual tag 1802 with fixed coordinates. A mobile asset may include anIoT sensor combined with a transceiver.

Referring now to FIG. 18B, an illustration is provided of a userinterface 1810 on a Smart Device 1812. The user interface includes imagedata 1810B representative of a physical environment 1800B in a directionof interest 1811 generated via the orientation of the smart device 1812by the user 1824. The user interface 1810 also includes an icon 1801associated with a physical asset in the area, as illustrated, the assetmay be a facility utility control 1814, other assets may be similarlyassociated with an icon. Some embodiments may also include aninteractive area 1813 that is congruent with an area and/or inclusive ofan area on the interactive interface of a physical environment 1800Bthat corresponds with an asset or other feature.

Similarly, the user may select the icon 1801 and activate presentationof digital data associated with the facility utility control 1814.

Referring now to FIG. 18C, the step of activating an icon 1821 via auser 1824 touch the interactive area including the icon 1821 isillustrated.

Referring now to FIG. 18D, an action screen 1823 that is generated inresponse to the user selecting the interactive area is illustrated. Theaction screen includes additional user interactive areas that areoperative with the controller to monitor a temperature sensor 1823A,monitor humidity 1823B, open and/or close an alarm state 1823C, and/ormonitor a CO2 meter 1823D.

Infrastructure Examples

Referring now to FIGS. 19, and 19A-19J, the elements referred to in anexemplary manner in FIG. 9A may be utilized in various manners forvarious infrastructure examples. The various methods and apparatusdisclosed in the present specification and the references made withinmay be applied variously to different types of infrastructure. In afirst example, as illustrated in FIG. 19 the infrastructure may be abridge. Various types of bridges, draw bridges, overpasses, raisedroadways and the like may have similar aspects to the illustrated cablestayed suspension bridge. In the illustration, a user may view a bridgefrom a remote location with a smart device 901. The smart device mayinteract with an array of transceiving nodes such as transceivers 908A-D and with the orienteering methods and equipment as described mayrecord a location and direction orientation of the user/smart device.The direction and location may be used by data processing apparatus tocreate overlays of functional elements that may be accessed by a user.Although the illustration depicts a user location at a remote point, thesame aspects may be applied when a user is close to an infrastructuresuch as bridge 1906 or upon it, or within it so long as the radiotransmissions with the transceivers may function. In some examples, theuser may orient their smart device to a desired direction of interestand then have a representation of the bridge 1904 displayed on thetouchscreen 902 of the smart device 901. There may be various iconssuperimposed upon the display such as icon 1900. In some examples theicons may represent the presence of a physical tag with various sensors1907 that can sense aspects of bridge function. Various sensing aspectshave been described in reference to FIG. 2C. The user may interact withthose sensors and understand their current values, their recordedhistory and/or controls of the sensing device may be accessed.

In some other examples, the icon may refer to a virtual tag associatedwith the bridge 1906. A virtual tag may be findable and accessible toauthorized users who send location and direction information to a servercontaining relevant information of the bridge 1906. In some examples,the virtual tag may give access to historical information related to thebridge 1906 including historical sensor results, designs, engineeringchange, engineering test results and the like. In other examples,perhaps in addition to this information, the virtual tag may give accessto communications of previous users who accessed the tag. In this sense,the virtual tag may utilize permanence concepts as discussed. In anexample, an employee of a utility that operates a bridge 1906 mayutilize his smart device to communicate a location and direction. Ininitial steps, the location of the user may be determined by modalitiessuch as GPS and cellular location protocols. When the region isdetermined further location and direction information may be determinedby an array of transceivers such as 908A-D. Identification informationof the user may be used to authorize access to server informationrelated to the bridge 1906. The user may be presented with a userinterface containing a number of tags that the user could access forinformation. In some examples, the interface may directly present theview of the bridge. In other examples, all relevant tags in theenvironment may be presented as in the example of FIG. 9B.

The user may select a particular tag or may reorient the direction ofthe smart device to find other tags in the environment. The user mayaccess a virtual tag and add reports, images, comments, observations,and the like. In a non-limiting example, the user may be performing aninspection of bridge components for wear, integrity, and the like. Theuser may record detailed inspection results and imagery to be stored onthe server associated with the virtual tag. In some examples, the servermay run application programs containing algorithms such as artificialintelligence or machine learning algorithms which may analyze thefinding and imagery for engineering analysis and assess the criticalityof the state of repair of the bridge components. Many other types ofinformation may be recorded, noted, or provided by the user.

Referring now to FIG. 19A, the infrastructure may be a dam, lock, orother water control structure. In the illustration, a user may view adam from a remote location with a smart device 901. The smart device mayinteract with an array of transceiving nodes such as transceivers 908A-D and with the orienteering methods and equipment as described mayrecord a location and direction orientation of the user/smart device.The direction and location may be used by data processing apparatus tocreate overlays of functional elements that may be accessed by a user.Although the illustration depicts a user location at a remote point, thesame aspects may be applied when a user is close to an infrastructuresuch as dam 1906A or upon it, or upon the water behind theinfrastructure, so long as the radio transmissions with the transceiversmay function. In some examples, the user may orient their smart deviceto a desired direction of interest and then have a representation of thedam 1904A displayed on the touchscreen 902 of the smart device 901.

There may be various icons superimposed upon the display such as icon1900A. In some examples, the icons may represent the presence of aphysical tag with various sensors 1907A that can sense aspects of damfunction. Various sensing aspects have been described in reference toFIG. 2D. The user may interact with those sensors and understand theircurrent values, their recorded history and/or controls of the sensingdevice may be accessed.

In some other examples, the icon may refer to a virtual tag associatedwith the dam 1906A. A virtual tag may be findable and accessible toauthorized users who send location and direction information to a servercontaining relevant information of the dam 1906A. In some examples, thevirtual tag may give access to historical information related to the dam1906A including historical sensor results, designs, engineering change,engineering test results and the like. In other examples, perhaps inaddition to this information, the virtual tag may give access tocommunications of previous users who accessed the tag. In this sense,the virtual tag may utilize permanence concepts as discussed. In anexample, an employee of a utility that operates a dam 1906A may utilizehis smart device to communicate a location and direction. In initialsteps, the location of the user may be determined by modalities such asGPS and cellular location protocols. When the region is determinedfurther, location and direction information may be determined by anarray of transceivers such as 908A-D. Identification information of theuser may be used to authorize access to server information related tothe dam 1906A. The user may be presented with a user interfacecontaining a number of tags that the user could access for information.In some examples, the interface may directly present the view of thedam. In other examples, all relevant tags in the environment may bepresented as in the example of 9B.

The user may select a particular tag or may reorient the direction ofthe smart device to find other tags in the environment. The user mayaccess a virtual tag and add reports, images, comments, observations,and the like. In a non-limiting example, the user may be performing aninspection of dam components for wear, integrity, and the like. The usermay record detailed inspection results and imagery to be stored on theserver associated with the virtual tag. In some examples, the server mayrun application programs containing algorithms such as artificialintelligence or machine learning algorithms which may analyze thefinding and imagery for engineering analysis and assess the criticalityof the state of repair of the dam components. Many other types ofinformation may be recorded, noted, or provided by the user.

In some additional examples, an analysis performed on engineering dataand design systems may determine structural aspects related toinformation retrieved from sensors and/or from measurements orrecordings of the user. The system may provide guidance for continuedmeasurement and analysis to be performed on the dam. In some examples,the user's smart device may be used to guide the user to an appropriatelocation and direction focus for continued observation and dataacquisition. The analysis may relate to various systems and subsystemsof the dam including abutments, arches, bearing surfaces, beams andstructural supports, decks, and other parts of the substructure andsuperstructure.

Referring now to FIG. 19B, the infrastructure may be a railway, subway,or other train conveyance structure. In the illustration, a user mayview a railway from a remote location with a smart device 901. The smartdevice may interact with an array of transceiving nodes such astransceivers 908 A-D and with the orienteering methods and equipment asdescribed may record a location and direction orientation of theuser/smart device. The direction and location may be used by dataprocessing apparatus to create overlays of functional elements that maybe accessed by a user. Although the illustration depicts a user locationat a remote point, the same aspects may be applied when a user is closeto an infrastructure such as railway 1906B or upon it, or upon the trainupon the railway, so long as the radio transmissions with thetransceivers may function. In some examples, the user may orient theirsmart device to a desired direction of interest and then have arepresentation of the railway 1904B displayed on the touchscreen 902 ofthe smart device 901.

There may be various icons superimposed upon the display such as icon1900B. In some examples, the icons may represent the presence of aphysical tag with various sensors 1907B that can sense aspects ofrailway function. Various sensing aspects have been described inreference to FIG. 2E. The user may interact with those sensors andunderstand their current values, their recorded history and/or controlsof the sensing device may be accessed.

In some other examples, the icon may refer to a virtual tag associatedwith the railway 1906B. A virtual tag may be findable and accessible toauthorized users who send location and direction information to a servercontaining relevant information of the railway 1906B. In some examples,the virtual tag may give access to historical information related to therailway 1906B including historical sensor results, designs, engineeringchange, engineering test results and the like. In other examples,perhaps in addition to this information, the virtual tag may give accessto communications of previous users who accessed the tag. In this sense,the virtual tag may utilize permanence concepts as discussed. In anexample, an employee of a utility that operates a railway 1906B mayutilize his smart device to communicate a location and direction. Ininitial steps, the location of the user may be determined by modalitiessuch as GPS and cellular location protocols. When the region isdetermined further, location and direction information may be determinedby an array of transceivers such as 908A-D. Identification informationof the user may be used to authorize access to server informationrelated to the railway 1906B. The user may be presented with a userinterface containing a number of tags that the user could access forinformation. In some examples, the interface may directly present theview of the railway. In other examples, all relevant tags in theenvironment may be presented as in the example of 9B.

The user may select a particular tag or may reorient the direction ofthe smart device to find other tags in the environment. The user mayaccess a virtual tag and add reports, images, comments, observations,and the like. In a non-limiting example, the user may be performing aninspection of railway components for wear, integrity, and the like. Theuser may record detailed inspection results and imagery to be stored onthe server associated with the virtual tag. In some examples, the servermay run application programs containing algorithms such as artificialintelligence or machine learning algorithms which may analyze thefinding and imagery for engineering analysis and assess the criticalityof the state of repair of the railway components. Many other types ofinformation may be recorded, noted, or provided by the user.

In some additional examples, an analysis performed on engineering dataand design systems may determine structural aspects related toinformation retrieved from sensors and/or from measurements orrecordings of the user. The system may provide guidance for continuedmeasurement and analysis to be performed on the railway. In someexamples, the user's smart device may be used to guide the user to anappropriate location and direction focus for continued observation anddata acquisition. The analysis may relate to various systems andsubsystems of the railway including tracks, rails, upstream anddownstream faces, spillways, and other parts of the substructure andsuperstructure.

Referring now to FIG. 19C, the infrastructure may be a roadway, highway,or other traffic conveyance structure. In the illustration, a user mayview a roadway from a remote location with a smart device 901. The smartdevice may interact with an array of transceiving nodes such astransceivers 908 A-D and with the orienteering methods and equipment asdescribed may record a location and direction orientation of theuser/smart device. The direction and location may be used by dataprocessing apparatus to create overlays of functional elements that maybe accessed by a user. Although the illustration depicts a user locationat a remote point, the same aspects may be applied when a user is closeto an infrastructure such as roadway 1906C or upon it so long as theradio transmissions with the transceivers may function. In someexamples, the user may orient their smart device to a desired directionof interest and then have a representation of the roadway 1904Cdisplayed on the touchscreen 902 of the smart device 901.

There may be various icons superimposed upon the display such as icon1900C. In some examples, the icons may represent the presence of aphysical tag with various sensors 1907C that can sense aspects ofroadway function. Various sensing aspects have been described inreference to FIG. 2F. The user may interact with those sensors andunderstand their current values, their recorded history and/or controlsof the sensing device may be accessed.

In some other examples, the icon may refer to a virtual tag associatedwith the roadway 1906C. A virtual tag may be findable and accessible toauthorized users who send location and direction information to a servercontaining relevant information of the roadway 1906C. In some examples,the virtual tag may give access to historical information related to theroadway 1906C including historical sensor results, designs, engineeringchange, engineering test results and the like. In other examples,perhaps in addition to this information, the virtual tag may give accessto communications of previous users who accessed the tag. In this sense,the virtual tag may utilize permanence concepts as discussed. In anexample, an employee of a utility that operates a roadway 1906C mayutilize his smart device to communicate a location and direction. Ininitial steps, the location of the user may be determined by modalitiessuch as GPS and cellular location protocols. When the region isdetermined further, location and direction information may be determinedby an array of transceivers such as 908A-D. Identification informationof the user may be used to authorize access to server informationrelated to the roadway 1906C. The user may be presented with a userinterface containing a number of tags that the user could access forinformation. In some examples, the interface may directly present theview of the roadway. In other examples, all relevant tags in theenvironment may be presented as in the example of 9B.

The user may select a particular tag or may reorient the direction ofthe smart device to find other tags in the environment. The user mayaccess a virtual tag and add reports, images, comments, observations,and the like. In a non-limiting example, the user may be performing aninspection of roadway components for wear, integrity, and the like. Theuser may record detailed inspection results and imagery to be stored onthe server associated with the virtual tag. In some examples, the servermay run application programs containing algorithms such as artificialintelligence or machine learning algorithms which may analyze thefinding and imagery for engineering analysis and assess the criticalityof the state of repair of the roadway components. Many other types ofinformation may be recorded, noted, or provided by the user.

In some additional examples, an analysis performed on engineering dataand design systems may determine structural aspects related toinformation retrieved from sensors and/or from measurements orrecordings of the user. The system may provide guidance for continuedmeasurement and analysis to be performed on the roadway. In someexamples, the user's smart device may be used to guide the user to anappropriate location and direction focus for continued observation anddata acquisition. The analysis may relate to various systems andsubsystems of the roadway including roadway surface, lines, signage, andother parts of the substructure and superstructure.

Referring now to FIG. 19D, the infrastructure may be a utility conduit.In the illustration, a user may view a utility conduit with a smartdevice 901. The smart device may interact with an array of transceivingnodes such as transceivers 908 A-D and with the orienteering methods andequipment as described may record a location and direction orientationof the user/smart device. The direction and location may be used by dataprocessing apparatus to create overlays of functional elements that maybe accessed by a user. Although the illustration depicts a user locationat a remote point, the same aspects may be applied when a user is closeto an infrastructure such as utility conduit 1906D or within it, so longas the radio transmissions with the transceivers may function. In someexamples, the user may orient their smart device to a desired directionof interest and then have a representation of the utility conduit 1904Ddisplayed on the touchscreen 902 of the smart device 901.

There may be various icons superimposed upon the display such as icon1900D. In some examples, the icons may represent the presence of aphysical tag with various sensors 1907D that can sense aspects ofutility conduit function. Various sensing aspects have been described inreference to FIG. 2G. The user may interact with those sensors andunderstand their current values, their recorded history and/or controlsof the sensing device may be accessed.

In some other examples, the icon may refer to a virtual tag associatedwith the utility conduit 1906D. A virtual tag may be findable andaccessible to authorized users who send location and directioninformation to a server containing relevant information of the utilityconduit 1906D. In some examples, the virtual tag may give access tohistorical information related to the utility conduit 1906D includinghistorical sensor results, designs, engineering change, engineering testresults and the like. In other examples, perhaps in addition to thisinformation, the virtual tag may give access to communications ofprevious users who accessed the tag. In this sense, the virtual tag mayutilize permanence concepts as discussed. In an example, an employee ofa utility that operates a utility conduit 1906D may utilize his smartdevice to communicate a location and direction. In initial steps, thelocation of the user may be determined by modalities such as GPS andcellular location protocols. When the region is determined further,location and direction information may be determined by an array oftransceivers such as 908A-D. Identification information of the user maybe used to authorize access to server information related to the utilityconduit 1906D. The user may be presented with a user interfacecontaining a number of tags that the user could access for information.In some examples, the interface may directly present the view of thedam. In other examples, all relevant tags in the environment may bepresented as in the example of 9B.

The user may select a particular tag or may reorient the direction ofthe smart device to find other tags in the environment. The user mayaccess a virtual tag and add reports, images, comments, observations,and the like. In a non-limiting example, the user may be performing aninspection of dam components for wear, integrity, and the like. The usermay record detailed inspection results and imagery to be stored on theserver associated with the virtual tag. In some examples, the server mayrun application programs containing algorithms such as artificialintelligence or machine learning algorithms which may analyze thefinding and imagery for engineering analysis and assess the criticalityof the state of repair of the utility conduits components. Many othertypes of information may be recorded, noted, or provided by the user.

In some additional examples, an analysis performed on engineering dataand design systems may determine structural aspects related toinformation retrieved from sensors and/or from measurements orrecordings of the user. The system may provide guidance for continuedmeasurement and analysis to be performed on the utility conduits. Insome examples, the user's smart device may be used to guide the user toan appropriate location and direction focus for continued observationand data acquisition. The analysis may relate to various systems andsubsystems of the utility conduits including tunnels, air systems, andother parts of the substructure and superstructure.

Referring now to FIG. 19E, the infrastructure may be an aqueductinfrastructure. In the illustration, a user may view an aqueduct from aremote location with a smart device 901. The smart device may interactwith an array of transceiving nodes such as transceivers 908 A-D andwith the orienteering methods and equipment as described may record alocation and direction orientation of the user/smart device. Thedirection and location may be used by data processing apparatus tocreate overlays of functional elements that may be accessed by a user.Although the illustration depicts a user location at a remote point, thesame aspects may be applied when a user is close to an infrastructuresuch as an aqueduct 1906E or upon it, or upon the water of theinfrastructure, so long as the radio transmissions with the transceiversmay function. In some examples, the user may orient their smart deviceto a desired direction of interest and then have a representation of theaqueduct 1904E displayed on the touchscreen 902 of the smart device 901.

There may be various icons superimposed upon the display such as icon1900E. In some examples, the icons may represent the presence of aphysical tag with various sensors 1907E that can sense aspects ofaqueduct function. Various sensing aspects have been described inreference to FIG. 2H. The user may interact with those sensors andunderstand their current values, their recorded history and/or controlsof the sensing device may be accessed.

In some other examples, the icon may refer to a virtual tag associatedwith the aqueduct 1906E. A virtual tag may be findable and accessible toauthorized users who send location and direction information to a servercontaining relevant information of the aqueduct 1906E. In some examples,the virtual tag may give access to historical information related to theaqueduct 1906E including historical sensor results, designs, engineeringchange, engineering test results and the like. In other examples,perhaps in addition to this information, the virtual tag may give accessto communications of previous users who accessed the tag. In this sense,the virtual tag may utilize permanence concepts as discussed. In anexample, an employee of a utility that operates an aqueduct 1906E mayutilize his smart device to communicate a location and direction. Ininitial steps, the location of the user may be determined by modalitiessuch as GPS and cellular location protocols. When the region isdetermined further, location and direction information may be determinedby an array of transceivers such as 908A-D. Identification informationof the user may be used to authorize access to server informationrelated to the aqueduct 1906E. The user may be presented with a userinterface containing a number of tags that the user could access forinformation. In some examples, the interface may directly present theview of the aqueduct. In other examples, all relevant tags in theenvironment may be presented as in the example of 9B.

The user may select a particular tag or may reorient the direction ofthe smart device to find other tags in the environment. The user mayaccess a virtual tag and add reports, images, comments, observations,and the like. In a non-limiting example, the user may be performing aninspection of aqueduct components for wear, integrity, and the like. Theuser may record detailed inspection results and imagery to be stored onthe server associated with the virtual tag. In some examples, the servermay run application programs containing algorithms such as artificialintelligence or machine learning algorithms which may analyze thefinding and imagery for engineering analysis and assess the criticalityof the state of repair of the aqueduct components. Many other types ofinformation may be recorded, noted, or provided by the user.

In some additional examples, an analysis performed on engineering dataand design systems may determine structural aspects related toinformation retrieved from sensors and/or from measurements orrecordings of the user. The system may provide guidance for continuedmeasurement and analysis to be performed on the aqueduct. In someexamples, the user's smart device may be used to guide the user to anappropriate location and direction focus for continued observation anddata acquisition. The analysis may relate to various systems andsubsystems of the aqueduct including parts of the substructure andsuperstructure.

Referring now to FIG. 19F, the infrastructure may be a waterwayinfrastructure. In the illustration, a user may view a waterway from aremote location with a smart device 901. The smart device may interactwith an array of transceiving nodes such as transceivers 908 A-D andwith the orienteering methods and equipment as described may record alocation and direction orientation of the user/smart device. Thedirection and location may be used by data processing apparatus tocreate overlays of functional elements that may be accessed by a user.Although the illustration depicts a user location at a remote point, thesame aspects may be applied when a user is close to an infrastructuresuch as a waterway 1906F or upon it, so long as the radio transmissionswith the transceivers may function. In some examples, the user mayorient their smart device to a desired direction of interest and thenhave a representation of the waterway 1904F displayed on the touchscreen902 of the smart device 901.

There may be various icons superimposed upon the display such as icon1900F. In some examples, the icons may represent the presence of aphysical tag with various sensors 1907F that can sense aspects of thewaterway function. Various sensing aspects have been described inreference to FIG. 2J. The user may interact with those sensors andunderstand their current values, their recorded history and/or controlsof the sensing device may be accessed.

In some other examples, the icon may refer to a virtual tag associatedwith the waterway 1906F. A virtual tag may be findable and accessible toauthorized users who send location and direction information to a servercontaining relevant information of the waterway 1906F. In some examples,the virtual tag may give access to historical information related to thewaterway 1906F including historical sensor results, designs, engineeringchange, engineering test results and the like. In other examples,perhaps in addition to this information, the virtual tag may give accessto communications of previous users who accessed the tag. In this sense,the virtual tag may utilize permanence concepts as discussed. In anexample, an employee of a utility that operates a dam 1906A may utilizehis smart device to communicate a location and direction. In initialsteps, the location of the user may be determined by modalities such asGPS and cellular location protocols. When the region is determinedfurther, location and direction information may be determined by anarray of transceivers such as 908A-D. Identification information of theuser may be used to authorize access to server information related tothe waterway 1906F. The user may be presented with a user interfacecontaining a number of tags that the user could access for information.In some examples, the interface may directly present the view of thewaterway. In other examples, all relevant tags in the environment may bepresented as in the example of 9B.

The user may select a particular tag or may reorient the direction ofthe smart device to find other tags in the environment. The user mayaccess a virtual tag and add reports, images, comments, observations,and the like. In a non-limiting example, the user may be performing aninspection of waterway components for wear, integrity, and the like. Theuser may record detailed inspection results and imagery to be stored onthe server associated with the virtual tag. In some examples, the servermay run application programs containing algorithms such as artificialintelligence or machine learning algorithms which may analyze thefinding and imagery for engineering analysis and assess the criticalityof the state of repair of the waterway components. Many other types ofinformation may be recorded, noted, or provided by the user.

In some additional examples, an analysis performed on engineering dataand design systems may determine structural aspects related toinformation retrieved from sensors and/or from measurements orrecordings of the user. The system may provide guidance for continuedmeasurement and analysis to be performed on the waterway. In someexamples, the user's smart device may be used to guide the user to anappropriate location and direction focus for continued observation anddata acquisition. The analysis may relate to various systems andsubsystems of the waterway including other parts of the substructure andsuperstructure.

Referring now to FIG. 19G, the infrastructure may be a hydropowergenerator infrastructure. In the illustration, a user may view ahydropower generator from a remote location with a smart device 901. Thesmart device may interact with an array of transceiving nodes such astransceivers 908 A-D and with the orienteering methods and equipment asdescribed may record a location and direction orientation of theuser/smart device. The direction and location may be used by dataprocessing apparatus to create overlays of functional elements that maybe accessed by a user. Although the illustration depicts a user locationat a remote point, the same aspects may be applied when a user is closeto an infrastructure such as a hydropower generator 1906G or upon it, orupon the water behind the infrastructure, so long as the radiotransmissions with the transceivers may function. In some examples, theuser may orient their smart device to a desired direction of interestand then have a representation of the hydropower generator 1904Gdisplayed on the touchscreen 902 of the smart device 901.

There may be various icons superimposed upon the display such as icon1900G. In some examples, the icons may represent the presence of aphysical tag with various sensors 1907G that can sense aspects ofhydropower generator function. Various sensing aspects have beendescribed in reference to FIG. 2K. The user may interact with thosesensors and understand their current values, their recorded historyand/or controls of the sensing device may be accessed.

In some other examples, the icon may refer to a virtual tag associatedwith the hydropower generator 1906G. A virtual tag may be findable andaccessible to authorized users who send location and directioninformation to a server containing relevant information of thehydropower generator 1906G. In some examples, the virtual tag may giveaccess to historical information related to the hydropower generator1906G including historical sensor results, designs, engineering change,engineering test results and the like. In other examples, perhaps inaddition to this information, the virtual tag may give access tocommunications of previous users who accessed the tag. In this sense,the virtual tag may utilize permanence concepts as discussed. In anexample, an employee of a utility that operates the hydropower generator1906G may utilize his smart device to communicate a location anddirection. In initial steps, the location of the user may be determinedby modalities such as GPS and cellular location protocols. When theregion is determined further, location and direction information may bedetermined by an array of transceivers such as 908A-D. Identificationinformation of the user may be used to authorize access to serverinformation related to the hydropower generator 1906G. The user may bepresented with a user interface containing a number of tags that theuser could access for information. In some examples, the interface maydirectly present the view of the hydropower generator. In otherexamples, all relevant tags in the environment may be presented as inthe example of 9B.

The user may select a particular tag or may reorient the direction ofthe smart device to find other tags in the environment. The user mayaccess a virtual tag and add reports, images, comments, observations,and the like. In a non-limiting example, the user may be performing aninspection of hydropower generator components for wear, integrity, andthe like. The user may record detailed inspection results and imagery tobe stored on the server associated with the virtual tag. In someexamples, the server may run application programs containing algorithmssuch as artificial intelligence or machine learning algorithms which mayanalyze the finding and imagery for engineering analysis and assess thecriticality of the state of repair of the hydropower generatorcomponents. Many other types of information may be recorded, noted, orprovided by the user.

In some additional examples, an analysis performed on engineering dataand design systems may determine structural aspects related toinformation retrieved from sensors and/or from measurements orrecordings of the user. The system may provide guidance for continuedmeasurement and analysis to be performed on the hydropower generator. Insome examples, the user's smart device may be used to guide the user toan appropriate location and direction focus for continued observationand data acquisition. The analysis may relate to various systems andsubsystems of the hydropower generator including parts of thesubstructure and superstructure.

Referring now to FIG. 19H, the infrastructure may be a nuclear powergenerator infrastructure. In the illustration, a user may view a nuclearpower generator from a remote location with a smart device 901. Thesmart device may interact with an array of transceiving nodes such astransceivers 908 A-D and with the orienteering methods and equipment asdescribed may record a location and direction orientation of theuser/smart device. The direction and location may be used by dataprocessing apparatus to create overlays of functional elements that maybe accessed by a user. Although the illustration depicts a user locationat a remote point, the same aspects may be applied when a user is closeto an infrastructure such as a nuclear power generator 1906H or withinit, so long as the radio transmissions with the transceivers mayfunction. In some examples, the user may orient their smart device to adesired direction of interest and then have a representation of thenuclear power generator 1904H displayed on the touchscreen 902 of thesmart device 901.

There may be various icons superimposed upon the display such as icon1900H. In some examples, the icons may represent the presence of aphysical tag with various sensors 1907H that can sense aspects ofnuclear power generator function. Various sensing aspects have beendescribed in reference to FIG. 2L. The user may interact with thosesensors and understand their current values, their recorded historyand/or controls of the sensing device may be accessed.

In some other examples, the icon may refer to a virtual tag associatedwith the nuclear power generator 1906H. A virtual tag may be findableand accessible to authorized users who send location and directioninformation to a server containing relevant information of the nuclearpower generator 1906H. In some examples, the virtual tag may give accessto historical information related to the nuclear power generator 1906Hincluding historical sensor results, designs, engineering change,engineering test results and the like. In other examples, perhaps inaddition to this information, the virtual tag may give access tocommunications of previous users who accessed the tag. In this sense,the virtual tag may utilize permanence concepts as discussed. In anexample, an employee of a utility that operates the nuclear powergenerator 1906H may utilize his smart device to communicate a locationand direction. In initial steps, the location of the user may bedetermined by modalities such as GPS and cellular location protocols.When the region is determined further, location and directioninformation may be determined by an array of transceivers such as908A-D. Identification information of the user may be used to authorizeaccess to server information related to the nuclear power generator1906H. The user may be presented with a user interface containing anumber of tags that the user could access for information. In someexamples, the interface may directly present the view of the nuclearpower generator. In other examples, all relevant tags in the environmentmay be presented as in the example of 9B.

The user may select a particular tag or may reorient the direction ofthe smart device to find other tags in the environment. The user mayaccess a virtual tag and add reports, images, comments, observations,and the like. In a non-limiting example, the user may be performing aninspection of the nuclear power generator components for wear,integrity, and the like. The user may record detailed inspection resultsand imagery to be stored on the server associated with the virtual tag.In some examples, the server may run application programs containingalgorithms such as artificial intelligence or machine learningalgorithms which may analyze the finding and imagery for engineeringanalysis and assess the criticality of the state of repair of thenuclear power generator components. Many other types of information maybe recorded, noted, or provided by the user.

In some additional examples, an analysis performed on engineering dataand design systems may determine structural aspects related toinformation retrieved from sensors and/or from measurements orrecordings of the user. The system may provide guidance for continuedmeasurement and analysis to be performed on the nuclear power generator.In some examples, the user's smart device may be used to guide the userto an appropriate location and direction focus for continued observationand data acquisition. The analysis may relate to various systems andsubsystems of the nuclear power generator including parts of thesubstructure and superstructure.

Referring now to FIG. 19J, the infrastructure may be a coal/gas power orWaste recycling power generator infrastructure. In the illustration, auser may view a coal/gas power generator from a remote location with asmart device 901. The smart device may interact with an array oftransceiving nodes such as transceivers 908 A-D and with theorienteering methods and equipment as described may record a locationand direction orientation of the user/smart device. The direction andlocation may be used by data processing apparatus to create overlays offunctional elements that may be accessed by a user. Although theillustration depicts a user location at a remote point, the same aspectsmay be applied when a user is close to an infrastructure such as acoal/gas power generator 1906J or within it, so long as the radiotransmissions with the transceivers may function. In some examples, theuser may orient their smart device to a desired direction of interestand then have a representation of the coal/gas power generator 1904Jdisplayed on the touchscreen 902 of the smart device 901.

There may be various icons superimposed upon the display such as icon1900J. In some examples, the icons may represent the presence of aphysical tag with various sensors 1907J that can sense aspects ofcoal/gas power generator function. Various sensing aspects have beendescribed in reference to FIG. 2M. The user may interact with thosesensors and understand their current values, their recorded historyand/or controls of the sensing device may be accessed.

In some other examples, the icon may refer to a virtual tag associatedwith the coal/gas power generator 1906J. A virtual tag may be findableand accessible to authorized users who send location and directioninformation to a server containing relevant information of the coal/gaspower generator 1906J. In some examples, the virtual tag may give accessto historical information related to the coal/gas power generator 1906Jincluding historical sensor results, designs, engineering change,engineering test results and the like. In other examples, perhaps inaddition to this information, the virtual tag may give access tocommunications of previous users who accessed the tag. In this sense,the virtual tag may utilize permanence concepts as discussed. In anexample, an employee of a utility that operates the coal/gas powergenerator 1906J may utilize his smart device to communicate a locationand direction. In initial steps, the location of the user may bedetermined by modalities such as GPS and cellular location protocols.When the region is determined further, location and directioninformation may be determined by an array of transceivers such as908A-D. Identification information of the user may be used to authorizeaccess to server information related to the coal/gas power generator1906J. The user may be presented with a user interface containing anumber of tags that the user could access for information. In someexamples, the interface may directly present the view of the coal/gaspower generator. In other examples, all relevant tags in the environmentmay be presented as in the example of 9B.

The user may select a particular tag or may reorient the direction ofthe smart device to find other tags in the environment. The user mayaccess a virtual tag and add reports, images, comments, observations,and the like. In a non-limiting example, the user may be performing aninspection of the coal/gas power generator components for wear,integrity, and the like. The user may record detailed inspection resultsand imagery to be stored on the server associated with the virtual tag.In some examples, the server may run application programs containingalgorithms such as artificial intelligence or machine learningalgorithms which may analyze the finding and imagery for engineeringanalysis and assess the criticality of the state of repair of thecoal/gas power generator components. Many other types of information maybe recorded, noted, or provided by the user.

In some additional examples, an analysis performed on engineering dataand design systems may determine structural aspects related toinformation retrieved from sensors and/or from measurements orrecordings of the user. The system may provide guidance for continuedmeasurement and analysis to be performed on the coal/gas powergenerator. In some examples, the user's smart device may be used toguide the user to an appropriate location and direction focus forcontinued observation and data acquisition. The analysis may relate tovarious systems and subsystems of the coal/gas power generator includingparts of the substructure and superstructure.

Physical world and virtual-world based imagery related to theenvironment of a user may be presented via a user interface that maydisplay on a Smart Device screen or other interactive mechanism, or insome embodiments, be presented in an augmented of virtual environment,such as via a VR or AR headset. The imagery displayed upon these devicesmay represent a composite of image data reflective of a real-world datastream as well as digitally added/superimposed image data from a virtualor digital source data stream. A user may be presented a typical imageas it would look to the user's eyes physically, upon which digitalshapes representing virtual “Tags” may be superimposed to represent thepresence of digital information that may be accessed by a user. In otherexamples, the digital information may be directly displayed as asuperposition. In some examples, the real-world and virtual-worldenvironments may be displayed separately on a screen or separately intime.

In some examples, the “physical world image” may also be digitallyformed or altered. For, example, an imaging device may obtain imageswhere the sensing elements of the imaging device are sensitive to adifferent frequency of electromagnetic radiation, such as in anon-limiting sense infrared radiation. The associated “real-world image”may be a color scale representation of the images obtained in theinfrared spectrum. In still further examples, two different real-worldimages may be superimposed upon each other with or without additionalvirtual elements. Thus, a sensor image may have an IR sensor imagesuperimposed over part or all of the image and a digital shaperepresenting a virtual Tag may be superimposed.

In some implementations, a virtual reality headset may be worn by a userto provide an immersive experience from a vantage point such that theuser may experience a virtual representation of what it would be like tobe located at the vantage point within an environment at a specifiedpoint in time. The virtual representation may include a combination ofsimulated imagery, textual data, animations and the like and may bebased on scans, image acquisition and other Sensor inputs, as examples.A virtual representation may therefore include a virtual representationof image data via the visual light spectrum, image data representingimage scans obtained via infrared light spectrum, noise, and vibrationreenactment. Although some specific types of exemplary sensor data havebeen described, the descriptions are not meant to be limiting unlessspecifically claimed as a limitation and it is within the scope of thisdisclosure to include a virtual representation based upon other types ofcaptured sensor data may also be included in the AVM virtual realityrepresentation.

It should be noted that although a Smart Device is generally operated bya human Agent, some embodiments of the present disclosure include acontroller, accelerometer, data storage medium, Image Capture Device,such as a CCD capture device and/or an infrared capture device beingavailable in an Agent that is an unmanned vehicle, including for examplean unmanned ground vehicle (“UGV”) such as a unit with wheels or tracksfor mobility and a radio control unit for communication. or an unmannedaerial vehicle (“UAV”) or other automation.

In some embodiments, multiple unmanned vehicles may capture data in asynchronized fashion to add depth to the image capture and/or athree-dimensional and four-dimensional (over time) aspect to thecaptured data. In some implementations, UAV position may be containedwithin a perimeter and the perimeter may have multiple reference pointsto help each UAV (or other unmanned vehicle) determine a position inrelation to static features of a building within which it is operatingand also in relation to other unmanned vehicles. Still other aspectsinclude unmanned vehicles that may not only capture data, but alsofunction to perform a task, such as paint a wall, drill a hole, cutalong a defined path, or other function. As stated throughout thisdisclosure, the captured data may be incorporated into a virtual modelof a space or Structure.

Glossary

“Agent” as used herein refers to a person or automation capable ofsupporting a Smart Device at a geospatial location relative to a GroundPlane.

“Area of digital content interaction” as used herein refers to an areawith multiple addition al sets of positional coordinates that may beused to access digital content with a first set of positionalcoordinates.

“Augmented Virtual Model” (sometimes referred to herein as “AVM”): asused herein is a digital representation of a real property parcelincluding one or more three-dimensional representations of physicalstructures suitable for use and As Built data captured descriptive ofthe real property parcel. An Augmented Virtual Model includes As BuiltFeatures of the structure and may include improvements and featurescontained within a Processing Facility.

“Bluetooth” as used herein means the Wireless Personal Area Network(WPAN) standards managed and maintained by Bluetooth SIG. Unlessotherwise specifically limited to a subset of all Bluetooth standards,the Bluetooth may encompass all Bluetooth standards (e.g., Bluetooth4.0; 5.0; 5.1 and BLE versions).

“Digital Content” as used herein refers to any artifact that may bequantified in digital form, By way of non-limiting example, digitalcontent may include, one or more of: alphanumeric text; audio files;image data; video data; digital stories and media.

“Energy-Receiving Sensor” as used herein refers to a device capable ofreceiving energy from a Radio Target Area and quantifying the receivedenergy as a digital value.

“Ground Plane” as used herein refers to horizontal plane from which adirection of interest may be projected.

“Hybrid Tag” as used herein means digital content associated with alocation coordinates of a position previously occupied by a PhysicalTag. In some embodiments, a Hybrid Tag may include digital content basedupon data generated by a sensor co-located with a Physical Tag or whilethe sensor was within a specified distance of a position described bylocation coordinates.

“Image Capture Device” or “Scanner” as used herein refers to apparatusfor capturing digital or analog image data, an Image capture device maybe one or both of: a two-dimensional sensor (sometimes referred to as“2D”) or a three-dimensional sensor (sometimes referred to as “3D”). Insome examples an Image Capture Device includes a charge-coupled device(“CCD”) sensor. “Intelligent Automation” as used herein refers to alogical processing by a device, system, machine, or equipment item (suchas data gathering, analysis, artificial intelligence, and functionaloperation) and communication capabilities.

“IoT Tag” as used herein refers to a Node co-located with an IoT Sensor.

“Multi-Modal” as used herein refers to the ability of a device tocommunication using multiple protocols and/or bandwidths. Examples ofmultimodal may include being capable of communication using two to moreof: Ultra-Wideband, Bluetooth; Bluetooth Low Energy; Wi-Fi; Wi-Fi RT;GPS; ultrasonic; infrared protocols and/or mediums.

“Multi-Modal Tag” as used herein refers to a device including multiplewireless transceivers operating in different bandwidths and according todifferent communication parameters.

“Node” as used herein means a device including at least a processor, adigital storage, and a wireless transceiver.

“Physical Tag” as used here shall mean a physical device with atransceiver capable of wireless communication sufficient to determine ageospatial position of the device. The Physical Tag may also beassociated with a data set that is not contingent upon the geospatiallocation of the physical device.

“Radio Target Area” an area from which an energy-receiving Sensor mayreceive energy of a type and bandwidth that may be quantified by theenergy-receiving Sensor.

“Ray” as used herein refers to a straight line including a startingpoint and extending indefinitely in a direction.

“Sensor” (sometimes referred to as an IoT sensor) as used herein refersto one or more of a solid state, electro-mechanical, and mechanicaldevice capable of transducing a physical condition or property into ananalogue or digital representation and/or metric.

“Smart Device” as used herein includes an electronic device including,or in logical communication with, a processor and digital storage andcapable of executing logical commands.

“Smart Receptacle” as used herein includes a case or other receiver of asmart device with components capable of receiving wireless transmissionsfrom multiple wireless positional reference transceivers. In someembodiments, the smart receptacle may include a wireless transmitterand/or a physical connector for creating an electrical path for carryingone or both of electrical power and logic signals between an associatedSmart Device and the Smart Receptacle.

“Structure” as used herein refers to a manmade assembly of partsconnected in an ordered way. Examples of a Structure in this disclosureinclude a building; a sub-assembly of a building; a bridge, a roadway, atrain track, a train trestle, an aqueduct; a tunnel a dam, and aretainer berm.

“Tag” as used herein refers to digital content and access rightsassociated with a geospatial position.

“Transceive” as used herein refers to an act of transmitting andreceiving data.

“Transceiver” as used herein refers to an electronic device capable ofone or both of wirelessly transmitting and receiving data.

“Vector” as used herein refers to a magnitude and a direction as may berepresented and/or modeled by a directed line segment with a length thatrepresents the magnitude and an orientation in space that represents thedirection.

“Virtual Tag” as used here shall mean digital content associated with alocation identified via positional coordinates.

“Wireless Communication Area” (sometimes referred to as “WCA”) as usedherein means an area through which wireless communication may becompleted. A size of a WCA may be dependent upon a specified modality ofwireless communication and an environment through which the wirelesscommunication takes place.

In discussion (and as illustrated), a WCA may be portrayed as beingspherical in shape, however in a physical environment a shape of a WCAmay be amorphous or of changing shape and more resemble a cloud ofthinning density around the edges.

A number of embodiments of the present disclosure have been described.While this specification contains many specific implementation details,there should not be construed as limitations on the scope of anydisclosures or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of the present disclosure.While embodiments of the present disclosure are described herein by wayof example using several illustrative drawings, those skilled in the artmay recognize the present disclosure is not limited to the embodimentsor drawings described. It should be understood the drawings and thedetailed description thereto are not intended to limit the presentdisclosure to the form disclosed, but to the contrary, the presentdisclosure is to cover all modification, equivalents and alternativesfalling within the spirit and scope of embodiments of the presentdisclosure as defined by the appended claims.

The headings used herein are for organizational purposes only and arenot meant to be used to limit the scope of the description or theclaims. As used throughout this application, the word “may” is used in apermissive sense (i.e., meaning having the potential to), rather thanthe mandatory sense (i.e., meaning must). Similarly, the words“include”, “including”, and “includes” mean including but not limitedto. To facilitate understanding, like reference numerals have been used,where possible, to designate like elements common to the figures.

The phrases “at least one”, “one or more”, and “and/or” are open-endedexpressions that are both conjunctive and disjunctive in operation. Forexample, each of the expressions “at least one of A, B and C”, “at leastone of A, B, or C”, “one or more of A, B, and C”, “one or more of A, B,or C” and “A, B, and/or C” means A alone, B alone, C alone, A and Btogether, A and C together, B and C together, or A, B and C together.

The term “a” or “an” entity refers to one or more of that entity. Assuch, the terms “a” (or “an”), “one or more” and “at least one” can beused interchangeably herein. It is also to be noted the terms“comprising”, “including”, and “having” can be used interchangeably.

Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented incombination in multiple embodiments separately or in any suitablesub-combination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

Similarly, while method steps may be depicted in the drawings in aparticular order, this should not be understood as requiring that suchoperations be performed in the particular order shown or in a sequentialorder, or that all illustrated operations be performed, to achievedesirable results.

Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented incombination in multiple embodiments separately or in any suitablesub-combination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

Moreover, the separation of various system components in the embodimentsdescribed above should not be understood as requiring such separation inall embodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order show, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous. Nevertheless, it will be understood thatvarious modifications may be made without departing from the spirit andscope of the claimed disclosure.

What is claimed is:
 1. A method for accessing digital content associatedwith a position in a wireless communication area in a region of aninfrastructure, the method comprising the steps of: a. establishing agross position determination with a location sensor of a Smart Device;b. determining the region comprising the infrastructure and the grossposition determination with an application program of the Smart Device;c. determining that the Smart Device has access to an array of referencepoint transceivers associated with the infrastructure and authorizingcommunications between the Smart Device and reference point transceiversof the array of reference point transceivers; d. transceiving a firstwireless communication between the Smart Device and a first referencepoint transceiver fixedly located at a first position within thewireless communication area; e. transceiving a second wirelesscommunication between the Smart Device and a second reference pointtransceiver fixedly located at a second position within the wirelesscommunication area; f. transceiving a third wireless communicationbetween the Smart Device and a third reference point transceiver fixedlylocated at a third position within the wireless communication area; g.generating positional coordinates for the Smart Device at an instance intime based upon the first wireless communication, the second wirelesscommunication, and the third wireless communication, each wirelesscommunication between the Smart Device and a respective one of the firstreference point transceiver, the second reference point transceiver, andthe third reference point transceiver; h. generating a direction ofinterest from the positional coordinates of the Smart Device; i.generating an area of interest comprising the direction of interest; j.generating positional coordinates of a tag at the instance in time; k.associating the digital content with the tag; l. determining that thepositional coordinates of the tag are within the area of interest; m.determining that the Smart Device has access rights to the digitalcontent; n. receiving a user input into a dynamic portion of a userinteractive interface on the Smart Device, the user input operative tocause the Smart Device to display the digital content; and o. based uponthe user input received into the dynamic portion of the user interactiveinterface, displaying the digital content related to the infrastructurein the user interactive interface.
 2. The method of claim 1 wherein thelocation sensor utilizes Global Position System radio signals todetermine the gross position determination.
 3. The method of claim 1wherein the location sensor utilizes Global Position System radiosignals and cellular network signals to determine the gross positiondetermination.
 4. The method of claim 1 wherein the location sensorutilizes cellular network signals alone to determine the gross positiondetermination.
 5. The method of claim 1 wherein the array of referencepoint transceivers comprises more than three reference pointtransceivers.
 6. The method of claim 5 wherein the array of referencepoint transceivers forms a self-verifying array of nodes.
 7. The methodof claim 6 wherein the tag is a physical tag comprising a sensormeasuring at least a first physical attribute of the infrastructure. 8.The method of claim 7 wherein the digital content comprises anhistorical record of measurements of the first physical attribute. 9.The method of claim 7 wherein a server comprising the digital contentperforms algorithmic calculations upon the measurements of the firstphysical attribute to generate a first message to a user as a portion ofthe digital content.
 10. The method of claim 9 wherein the first messageto the user provides directions for the user to move to a fourthposition, wherein at the fourth position the user records a secondmeasurement with an energy receiving sensor of the Smart Device torecord an image of a state of the infrastructure.
 11. The method ofclaim 10 wherein the infrastructure is a bridge.
 12. The method of claim6 wherein the tag is a virtual tag, wherein the virtual tag associates acommunication node with the infrastructure.
 13. The method of claim 12wherein the digital content comprises persistent data associated withthe virtual tag.
 14. The method of claim 13 wherein the digital contentcomprises a second message from a second user to users who gain accessto the virtual tag.
 15. The method of claim 12 wherein the Smart Devicecomprises a connected sensor device, wherein the connected sensor deviceproduces a measurement of a second physical attribute of theinfrastructure and wherein the Smart Device communicates data associatedwith the measurement of the second physical attribute as persistent datato be stored associated with the virtual tag.
 16. The method of claim 15wherein a server comprising the digital content performs algorithmiccalculations upon the measurements of the second physical attribute togenerate a second message to the user as a portion of the digitalcontent.
 17. The method of claim 16 wherein the second message to theuser provides directions for the user to move to a fifth position,wherein at the fifth position the user records a third measurement withan energy receiving sensor of the Smart Device to record an image of astate of the infrastructure.
 18. The method of claim 17 wherein theinfrastructure is a bridge.
 19. The method of claim 12 wherein the useris presented with a first user interface of the Smart Device, whereinthe tag is presented as an icon overlaid on an image generated based onthe position and the direction of interest of the user.
 20. The methodof claim 19 wherein the first user interface presents a set of iconseach icon at the set of icons presented at a respective location valueof both virtual tags and physical tags in the region of theinfrastructure.